CS 159

Lab 4: Part of Speech Tagging

Right now, our code is set up to use the Penn TreeBank (PTB) tag set. This is a commonly-used set of part of speech tags for English, and you can find the full set in the textbook (Figure 17.2, on page 4 of Chapter 17). But this is not the only definition of parts of speech that exists. One other commonly-used alternative is the Universal Dependencies tag set, meant to be a minimalist but generalizable (across multiple languages) set of part of speech tags. You can see the listing of Universal Dependencies tags on their website. Given that Universal Dependencies is a less granular tag set than Penn TreeBank, we might wonder if it has an impact on how "difficult" the task of part of speech tagging is. That's the question we'll explore in this part of the lab!

To start, open up read_tags.py. At the top of the file, there is a dictionary called universal_to_ptb. The keys of this dictionary are Universal Tags, while the values are tuples of PTB tags that correspond to that tag. For instance, the tag "VERB" in the Universal Tag Set encompasses the tags "MD" (for modal verbs) as well as a number of tags starting with a V representing different verb forms (e.g. non-3rd person, gerunds, etc).

Your coding task for this part of the lab is to add support for Universal Dependencies tags in the code, as an alternative to the Penn TreeBank tags. To do this, you will go through the following steps:

  1. Add code to create a dictionary ptb_to_universal. This dictionary will have a key for each PTB tag, whose value will be the corresponding Universal tag. Values in this dictionary should just be strings, not lists of strings. While you can do this by just typing in the whole dictionary, it will be much easier (and better style!) to write a small snippet of code to generate the ptb_to_universal dictionary from the existing universal_to_ptb.
  2. Using this dictionary, modify the parse_file function to convert tags from PTB tags to Universal tags if do_universal is True.
  3. Add a new flag universal to the interface for the HmmTagger.py file in the Argument Parser. You should keep the existing default functionality if this flag isn't used, so have universal default to False by making its action="store_true". This will make it so calling the code with --universal will enable using this conversion, and not using the flag will default to the Penn TreeBank.
  4. Add a new named parameter do_universal to the HMMTagger class’s constructor. do_universal should default to False. Inside the HMMTagger object, save the new parameter as a data member called self.do_universal.
  5. Modify the HMMTagger constructor so that, when do_universal is true, the self.tags list is initialized with the Universal Dependencies tag set instead of the Penn TreeBank tag set.
  6. Update the main() function in HmmTagger.py to pass the new command-line argument args.universal to the HMMTagger initializer.
  7. Modify the train() function in HmmTagger.py to pass the value of self.do_universal to the parse_dir function. This will use the code you wrote for parse_file.
  8. Add a universal flag to evaluate.py just like you did for HmmTagger.py.
  9. Modify the main function of evaluate.py to pass the value of args.universal to parse_dir, just like you did for the training method of HMMTagger.

Once you are done with this process, use your newly modified code to answer the following analysis question:

Analysis Question #2

You will repeat the experiment from Analysis Question #1, but this time using the Universal Dependenices tags (by specifying your newly-added universal flag when running both scripts). Before you run any code, write down your hypothesis in your journal: do you think that using the Universal Dependencies tags will result in better accuracy, worse accuracy, or no meaningful effect? Justify your hypothesis. Now, run the experiment and report the accuracy that you get. Does the result match your hypothesis? If it does not, or if it's surprising in any other ways, briefly speculate on why.

Now, make one further adjustment to your code: we want to be able to train on the Penn TreeBank tags but test on the Universal Dependencies tags. To enable this, modify the main function of evaluate.py so that after it calls tagger, but before it compares the tags, it changes each token’s tag_ attribute to the right Universal Tag Set tag. If you leverage what you have imported from read_tags.py, this should only require a couple of lines of code. Then, use your newly modified code to answer the following analysis question:

Analysis Question #3

Run the experiment again with this new configuration, training on Penn TreeBank tags and testing on Universal Dependencies tags. Report the accuracy that you get. Which previous result is the right one to compare to: the result from Analysis Question #1, or the one from Analysis Question #2? How does it compare?

For the remainder of the lab, you should train using the full Penn TreeBank Tag Set and test using the Universal Dependencies Tag Set (i.e., use the same configuration as Analysis Question #3).

(When logged in, completion status appears here.)