CS 159

Lab 4: Part of Speech Tagging

Just like in Lab 3, the starter code includes a lot of functionality already implemented for you. In fact, this week's starter code contains a near-complete implementation of the Viterbi algorithm!

The implementation is split across three files:

  • HmmTagger.py defines a class HMMTagger that implements a Hidden Markov Model (HMM) part of speech tagger. You can find most of the Viterbi implementation in the predict function, with one piece missing for you to fill in soon.
  • evaluate.py runs the part of speech tagger on a corpus, and calculates the tagger's accuracy.
  • read_tags.py contains helper functions for loading a corpus that contains part of speech labels.

HMMTagger usage

The two main externally-facing functions of an HMMTagger object are train and predict. You can use an HMMTagger like this:

nlp = spacy.load("en_core_web_sm")

train_dir = "/courses/cs159/data/pos/wsj/train"
tagger = HMMTagger(nlp, alpha=0.1)
tagger.train(train_dir)

test_sentence = nlp("This is test input to the Part of Speech Tagger.")
tagger.predict(test_sentence)
print([token.tag_ for token in test_sentence])
  • LParrot speaking

    Of course, if you try to run this right now, you will get nonsense results since the implementation is not yet complete.

  • RParrot speaking

    Once you finish part 2, running this snippet should get you more reasonable looking results!

HMMTagger is designed to integrate with spaCy, and implements its standard tagger interface. This means you can also treat an HMMTagger object like a function and "call" it directly instead of explicitly saying predict. This is just "syntactic sugar" and does not actually change the behavior, so you can use whichever syntax you prefer:

# This snippet does EXACTLY the same thing as the previous one!
nlp = spacy.load("en_core_web_sm")

train_dir = "/courses/cs159/data/pos/wsj/train"
tagger = HMMTagger(nlp, alpha=0.1, vocab_size=20000)
tagger.train(train_dir)

test_sentence = nlp("This is test input to the Part of Speech Tagger.")
tagger(test_sentence)
print([token.tag_ for token in test_sentence])

The HmmTagger.py file can also be run as a script from the command line. When you do that, it has the following interface:

usage: HmmTagger.py [-h] --dir DIR --output FILE [--alpha ALPHA]

Train (and save) hmm models for POS tagging

optional arguments:
  -h, --help            show this help message and exit
  --dir DIR, -d DIR     Read training data from DIR
  --output FILE, -o FILE
                        Save output to FILE
  --alpha ALPHA, -a ALPHA
                        Alpha value for add-alpha smoothing

Running HmmTagger.py as a script will train an HMMTagger object on all of the files in dir, then save the model for future reuse in pickled form. So for example:

python3 HmmTagger.py --dir /courses/cs159/data/pos/wsj/train --output model.pkl

Will train an HMMTagger on the wsj training set, and save the resulting HMMTagger object to the file model.pkl.

Evaluation script usage

Once you have trained an HMMTagger using the HmmTagger.py script, you can run evaluate.py to test the model on a test set and compute the accuracy. The evaluate.py script has the following interface:

usage: evaluate.py [-h] --dir DIR --hmm FILE

POS Tag, then evaluate

options:
  -h, --help     show this help message and exit
  --dir, -d DIR  Read data to tag from DIR
  --hmm FILE     Read hmm model from FILE

So for example, to run the trained model previously saved as model.pkl on the wsj test set, we would run:

python evaluate.py -d /courses/cs159/data/pos/wsj/test --hmm model.pkl

TASK: Comment HmmTagger.py

Take a moment now to familiarize yourself with the code in HmmTagger.py. Just like last week, your first task is to add comments to HmmTagger.py, including docstrings for every function. Once again, we will be reading your comments and they will contribute to your completeness and correctness scores! Therefore, you should aim to write comments that prove to us that you fully understand what the code is doing.

Also just like last week, you may encounter Python features or library calls that are unfamiliar to you (especially if you're not familiar with numpy). In particular, notice that we import the following functions from numpy:

from numpy import argmax, zeros, array, float32, ones, zeros, log

If any of these look unfamiliar to you, you may want to take a moment to look them up in the numpy documentation!

In addition, you'll see two matrices populated in the train function. These correspond to the emission and transition probability matrices we defined on the previous page. You should replace the TODO comment with information on which matrix is which.

There is no analysis question for this part of the assignment; your comments are the main deliverable. In addition, keep in mind that if you encounter syntax/functions that are unfamiliar to you and need to do some research to figure them out, that is something you should explicitly discuss in your journal!

(When logged in, completion status appears here.)