Lab 3: Text Classification

To run prediction on the Patronizing Data task, you’ll set up code in the pcl_main.py program. As usual, you can get a summary of the how to use this code (i.e., what arguments are accepted and what they mean) by running the program with the -h flag:

$ python3 pcl_main.py -h
usage: pcl_main.py [-h] [-o FILE] [-v N] [-s N] [--train_size N] (-t TEST_CATEGORY | -x XVALIDATE) data_file vocabulary

positional arguments:
  data_file             Data file containing labeled training instances
  vocabulary            File containing vocabulary words

options:
  -h, --help            show this help message and exit
  -o, --output_file FILE
                        Write predictions to FILE
  -v, --vocab_size N    Only count the top N words from the vocab file
  -s, --stop_words N    Exclude the top N words as stop words
  --train_size N        Only train on the first N instances. N=0 means use all training instances.
  -t, --test_category TEST_CATEGORY
  -x, --xvalidate XVALIDATE

Once it has parsed the command-line arguments, pcl_main calls the function do_experiment, which has not yet been implemented. Your task now is to implement do_experiment, following this outline:

1. Create an instance of PCLVocab.
2. Create an instance of MyFeatures.
3. Create an instance of BinaryLabels and CategoryLabels.
   • Note: the features and labels are all read from the same file, stored in the data_file argument. You will therefore end up reading from this file multiple times. To do this, you will have to call data_file.seek(0) to "reset" your position to the top of the file in between the constructor calls for MyFeatures, BinaryLabels, and CategoryLabels.
4. Create an instance of scikit-learn's MultinomialNB Naive Bayes classifier.
5. Using the process method of MyFeatures and BinaryLabels, create feature and target (X and y) matrices from the full data. Also extract the category of each example using the process method of CategoryLabels.
6. Depending on the value of args.xvalidate or args.test_category (which are mutually exclusive), either:
   • If a test category is given, then you'll use all of the examples from that category as test data, keeping only the examples without that category as training data. (You can use numpy array slicing or np.where to help do this.) You should fit your model to the data and labels for examples outside the category test_category, then get predictions (and probabilities) for each example matching the test category.
   • If a number of folds x is given, perform x-fold cross validation on the full data, getting predictions (and probabilities) for every example. Hint: It may help to inspect what the "method" parameter does in scikit-learn's cross_val_predict function!
7. Regardless of which method was used to generate predictions, write out one line to args.output_file for each prediction. The line should have three values in order, separated by spaces:
   • The example's id
   • The predicted class, true or false (do not include the quotes around the string)
   • your model’s confidence—that is, the probability of the predicted class

Please don't alter the order or format of the output file from the specification above, as semeval-pcl-2022-eval.py expects this format.

Hint: When writing the output to args.output_file, keep in mind that PCLLabels converts all labels to integers, so instead of "true" and "false", you will have 0 and 1. Furthermore, the correspondence between original labels and their integer values may not be what you expect! To find out what the correspondence is, PCLLabels has already implemented a __getitem__ method so you can use it like a dict. For example, if you have a BinaryLabels instance named binary_labeler, you can write binary_labeler[0] to find out what (string) label the integer 0 corresponds to.

Hint: If you're not sure how to get the model's confidence, you can start by looking at the predict_proba method of MultinomialNB. But you should note that predict_proba won't directly give you what you want (which is a single probability for the predicted class), and there's some additional work you'll need to do!

(When logged in, completion status appears here.)