Lab 3: Text Classification
Now that you have implemented do_experiment, let's run some experiments! For all of the following analysis questions, you should be using the following experiment settings unless otherwise stated:
- Vocabulary size of 10,000 words
- 10-fold cross validation
- All other settings should be left blank/default
You can use the following command to run pcl_main.py with the above settings (outputting to a file called testrun.hyp, though you can choose to name the output whatever you want):
python pcl_main.py /courses/cs159/data/patronize/patronize_full.xml /courses/cs159/data/patronize/vocab.txt -o testrun.hyp -v 10000 -x 10
pcl_main.py will save its predictions to the file specified by the -o argument (in the above example, testrun.hyp). We provide a separate script, semeval-pcl-2022-eval.py, that reads these predictions and computes standard evaluation metrics (like precision, recall, F1, and accuracy). To compute evaluation metrics, you can run the following command:
python semeval-pcl-2022-eval.py -d /courses/cs159/data/patronize/patronize_full.xml -r testrun.hyp
Analysis Question #3
Run the experiment using the above settings, and in your journal, paste the output of semeval-pcl-2022-eval.py showing all the evaluation metrics. Then, tell us what you think about these results. Are you satisfied? Unsatisfied? Based on these results, would you consider this model to be a "good" classifier?
Analysis Question #4
Now, replace the MultinomialNB Naive Bayes Classifier with a scikit-learn DummyClassifier. As the name implies, a DummyClassifier doesn't actually do any machine learning, and instead makes "dummy" predictions using simple heuristics. You can choose which heuristic to use by changing the strategy parameter in the DummyClassifier constructor. Try re-running the experiment with a DummyClassifier, trying at least two different heuristic strategies. Once gain, paste the evaluation metrics in your journal. Seeing the results of the DummyClassifier, does this change your answer to Analysis Question #3? (i.e., does it make you more satisfied or less satisfied with the Naive Bayes model)? Why or why not?
(Remember to change your code back to using MultinomialNB after you're done with this question!)
Note: When you run the experiments with the DummyClassifier, you may want to write the output to a different file, so that you don't overwrite the results of the original Naive Bayes experiment! We will still need those results for subsequent analysis questions.
Analysis Question #5
From the original Naive Bayes classifier output, identify (by id) three examples that your model is confident are condescending (i.e., the model predicts the example is condescending, and the assigned probability is high / close to 1.0). Comment on the contents of the examples: What do you think makes your classifier so confident? Is your classifier right?
Now answer the same questions but for three examples that your model is confident are not condescending.
Analysis Question #6
From the original Naive Bayes classifier output, identify (by id) three examples that your model is not confident about—that is, examples for which the classifier's assigned probability is very close to 0.5. Comment on their contents: what do you think makes these examples hard for your classifier? Do you find them hard to classify as a human? If not, what aspects of the examples do you take into account that are not captured by the features available to your classifier?
Analysis Question #7
Now, run the Naive Bayes experiment again, but this time instead of using 10-fold cross validation, try holding out the vulnerable category. This can be done by changing the -x 10 argument to -t vulnerable. All other setting should be kept unchanged.
From the resulting output, identify (by id) three vulnerable examples that your model is not confident about. Comment on their contents: what do you think makes these examples hard for your classifier? Do you find them hard to classify as a human? If not, what aspects of the examples do you take into account that are not captured by the features available to your classifier?
Based on your answers to the above, comment on whether this performed better or worse than cross validation. Did you find this result surprising?
(When logged in, completion status appears here.)