Lab 4: Part of Speech Tagging

The starter HMM model adds all of the words in the training set to its vocabulary. For the Brown data set, that means it has a vocabulary of 47,703 words, plus the <<OOV>> or "Out Of Vocabulary" special token.

In this section of the lab, you will explore some of the time, space, and performance trade-offs that come from varying the size of the vocabulary.

Your coding task for this part of the assignment is to modify the HmmTagger.py interface so that it can take a vocabulary size as a command-line argument. To do that, you should:

1. Add a --vocabsize, -v argument to the HmmTagger.py interface. The default value should be None, which will correspond to keeping all of the words in the vocabulary (in other words, the default behavior will be the same as the starter code behavior).
2. Modify update_vocab so that it only keeps args.vocabsize number of words from the vocabulary. Words should be picked in order of frequency, so the words that are kept are the most frequent words. If vocabsize is None, this behavior should be disabled and all words should be kept.

Now, train six models. All of them should be trained on the Brown corpus using the PTB tag set. They should vary in their vocabulary size: 1000, 2000, 5000, 10000, 20000, or 50000. Note that since 50000 is greater than the number of different word types in the Brown corpus, it should keep all of the tokens in the vocabulary (i.e., the behavior of vocabsize=50000 on this corpus should be identical to the behavior of vocabsize=None). You will use these models to answer the following analysis question:

Analysis Question #4

For each of the six models, report in your journal: - How long it took to train the model on /courses/cs159/data/pos/brown (in seconds) - How long it takes to test the model on the /courses/cs159/data/pos/wsj data (in seconds) - How big the model is (in kilobytes or megabytes, as appropriate) - The model's accuracy

Comment on any patterns you observe. Do the results surprise you or are they what you expected? Why?

Tip: The command-line time command can be used to time how long another program takes to run. For example, to see how long it takes to build model.pkl:

jpchang@chrysanthemum:~$ time python3 HmmTagger.py -d /courses/cs159/data/pos/brown -o model.pkl

real    1m13.679s
user    1m12.865s
sys     0m0.487s

This output says that it took 1:13 in real (clock) time, 1:12 in processor time (aka user time), and 0.487 seconds of system time to run HmmTagger.py. You should report the processor (user) time for this lab.

Tip: To find the size of a file, you can run ls -lh in the directory where the file is stored. For example:

jpchang@chrysanthemum:~$ ls -lh
total 39M
-rw-rw-r-- 1 jpchang jpchang 1.3K Oct  1 11:20 analysis.md
-rw-rw-r-- 1 jpchang jpchang  39M Oct  1 11:44 brown.pkl
-rwxrwxr-x 1 jpchang jpchang 1.3K Oct  1 11:50 evaluate.py
-rwxrwxr-x 1 jpchang jpchang 5.3K Oct  1 11:29 HmmTagger.py
drwxrwxr-x 2 jpchang jpchang 4.0K Oct  1 11:45 __pycache__
-rw-rw-r-- 1 jpchang jpchang 2.4K Oct  1 11:20 read_tags.py

Note the number next to the timestamp (date) of the file brown.pkl. It says 39M. This means brown.pkl is 39 megabytes in size.

(When logged in, completion status appears here.)