CS 159

Lab 3: Text Classification

As you were exploring the results and answering the analysis questions, you hopefully started to gain an intuition for what important aspects of the problem are not being captured by simple bag-of-words features. Now is your chance to try to make the classifier better! You will do this by changing the features that are extracted from the data. Remember that, as we discussed in class on Week 4 and practiced in the in-class activity, features can be more than just individual words; you can extract any information you think is important from the example, and add it to the list of features!

You will proceed by modifying the _extract_features function of your MyFeatures class to add (or remove!) features to the returned list of features. For example, suppose you wanted to extract a feature "does this text contain any all-uppercase words?". The way you could do that is:

  1. Split the text by whitespace without lowercasing.
  2. Loop through all the resulting tokens, and use Python's isupper to check if any of them are all uppercase.
  3. If any words were all uppercase, append a new feature to your list of features with a meaningful name like "CONTAINS_UPPERCASE". If none of the words were all uppercase, do not append this new feature.

This is just one example (and you don't have to use it), but hopefully it gives you an idea of what it means to add a new feature to the list of features. This part of the lab is open ended, and we encourage you to exercise your creativity! Your goal is to come up with, and implement, any additional features you can think of to (hopefully) improve the performance (accuracy, F1, etc) of the classifier.

As you come up with new features, be sure to document your thought process in your journal—even if those features don't end up actually helping! Every change you make to the feature set should be described and justified in your journal, otherwise you will lose points!

Here are some additional suggestions to get you started.

  • In addition to adding new features, you can also consider removing features. One way to do this without modifying _extract_features is to edit the vocabulary. In particular, the pcl_main.py script contains an argument you haven't used yet, stop_words. If you supply this argument, it will tell the vocabulary to skip past the N most common words (where N is the number you supply as the argument). If you use this option, you should justify in your writeup why you think it would help (as a hint, think back to your analysis of word counts in Lab 1!)
  • Furthermore, besides adding or removing features, you could also modify existing features! For example, maybe you think that the sequence "it is" should be considered the same as the token "it's". One way to achieve this is by scanning through your list of features, and if you see "it" followed by "is", remove them from the list and replace them with "it's". You could also do the opposite, replacing "it's" with "it" and "is".
  • Remember that a feature doesn't have to correspond to an actual word! Again: features can be anything. This can mean adding "symbolic" binary features like the "CONTAINS_UPPERCASE" example. Or, it could mean replacing actual words with "pseudo-words" as a way of approximating differences in meaning. For example, maybe you want to treat words inside quotations differently. You could mark such words by, for example, adding a prefix to them like "QUOTE-", so that the word "hello" inside a quote becomes "QUOTE-hello".
  • For this part of the lab, you can also change how you do tokenization, if you want. Note that if you do this it may cause some of the tests to start failing, but that's okay (we don't grade you using the tests anyway).

Once you are happy with the list of features, rerun the experiment and paste the final results in your journal. Then, answer the following analysis questions:

Analysis Question #8

Did your changes result in an improvement to the classifier performance? If so, was it as much of an improvement as you'd wanted? If not, do you observe any other changes / tradeoffs? Overall, are you satisfied with the performance of this classifier?

Analysis Question #9

Identify (by id) three examples that the classifier classifies differently now with your new features than it did before (e.g., the original classifier predicted condescending but not it predicts not condescending). Which (if any) of your modified features do you think made the difference? Do you think this change was a good thing? Does it better capture your intuitions as a human?

(When logged in, completion status appears here.)