Lab 1: Tokenization and Segmentation

We've built some functions for (very basic) tokenization—now let's do something interesting with them! For this part, you will explore some files from Project Gutenberg, a library of free eBooks for public domain texts.

We have already provided the files for you in a convenient, plain-text format. They can be found on the course server in the directory /courses/cs159/data/gutenberg/.

Part 2a: Exploring the Corpus

In the Gutenberg data directory there are a number of .txt files containing texts found in the Project Gutenberg collection. First, you should load the text of Lewis Carroll’s Alice's Adventures in Wonderland, which is stored in the file carroll-alice.txt. Use your tokens_by_frequency and count_tokens functions from Part 1 to explore the text. For the rest of this lab, you will always lowercase when getting a list of tokens. You should find that the five most frequent tokens in the text are:

the      1603
and       766
to        706
a         614
she       518

As an additional check: if your count_tokens function is working as we originally described it, it should report that the token "alice" occurs 221 times. Confirm that you get this result with your code. Now, whether this result is "correct" is a much more debatable topic, and I encourage you to reflect on it a little before continuing!

Note: If your numbers were right in the previous part, but don’t match here, it may be because of how you’re calling str.split. Take a look at the documentation for str.split to see if there’s a different way you can call it.

Analysis Question 1

You may have noticed that the most frequent tokens look a little...boring. They're basically just common connective and filler words like "the" and "and"! How far down the list do you have to go before you find an interesting token? Here, "interesting" is subjective and is entirely up to you, but be sure to at least give some justification for what you consider to be "interesting". What does this imply for someone who's trying to use this code for, say, a digital humanities project analyzing the writing style of Lewis Carroll? Are there any simple changes you might recommend to make the code more useful for that use case?

Reminder: For any lab in this course, if a question (like the one above) asks you to discuss results, that always means both what the results were and what that implies about the world (i.e., your corpus, your method, etc.). A sadly common way to lose points in this class has been forgetting to interpret or analyze results. A good recipe for full points on this sort of question is a paragraph that goes something like:

"The result was X...specific interesting examples were X' and X"...this is/isn't surprising because it would imply P or Q...to address this it might be better to do Y / to evaluate Z to learn more"

Even if your hypothesis of what's going on turns out to be different from the truth, you'll usually get full points for a plausible and descriptive answer that engages with concepts in NLP.

Part 2b: Handling Punctuation

There is a deficiency in how we implemented the get_tokens function. When we are counting tokens, we probably don’t care whether the token was adjacent to a punctuation mark. For example, the token "hatter" appears in the text 57 times, but if we queried the count_tokens dictionary, we would see it only appeared 24 times. However, it also appeared numerous times adjacent to a punctuation mark, so those instances got counted separately:

>>> token_freq = tokens_by_frequency(tokens)
>>> for (token, freq) in token_freq:
...     if 'hatter' in token:
...         print('{:10} {:3d}'.format(token, freq))
...
hatter      24
hatter.     13
hatter,     10
hatter:      6
hatters      1
hatter's     1
hatter;      1
hatter.'     1

Our get_tokens function would be better if it separated punctuation from tokens. We can accomplish this by using regular expressions! We will be making use of re, Python's standard regex library, and more specifically the re.split function. Be sure to add import re at the top of your file so you can access the re library functions. Below is a small example that demonstrates how str.split works on a small text and compares it to using re.split:

>>> text = '"Oh no, no," said the little Fly, "to ask me is in vain."'
>>> text.split()
['"Oh', 'no,', 'no,"', 'said', 'the', 'little', 'Fly,', '"to', 'ask', 'me', 'is',
 'in', 'vain."']
>>> re.split(r'(\W)', text)
['', '"', 'Oh', ' ', 'no', ',', '', ' ', 'no', ',', '', '"', '', ' ', 'said', ' ', 'the',
 ' ', 'little', ' ', 'Fly', ',', '', ' ', '', '"', 'to', ' ', 'ask', ' ', 'me', ' ', 'is',
 ' ', 'in', ' ', 'vain', '.', '', '"', '']

Note that this is not exactly what we want, but it is a lot closer. In the resulting list, we find empty strings and spaces, but we have also successfully separated the punctuation from the rest of the tokens.

Using the above example as a guide, write and test a function called tokenize that takes a string as an input and returns a list of tokens where punctuation has been separated out, and extraneous spaces and empty strings are not included. Like get_tokens, tokenize should take an optional argument do_lower that determines whether the string should be case normalized before separating the tokens. You don't need to come up with a new regex: just stick with (\W) and remove the empty strings and spaces after the re.split call.

To double check if things are working right, use your tokenize function in conjunction with your count_tokens function to list the top 5 most frequent tokens in carroll-alice.txt. You should get this:

'        2871      <-- single quote
,        2418      <-- comma
the      1642
.         988      <-- period
and       872

Part 2c: filter_nonwords

You may notice that right now, tokenize is returning punctuation marks as standalone tokens. For many applications (including Part 3), this is exactly what we want. But there may be other applications where we don't want to see punctuation in the list of tokens (can you think of some examples?).

Write a function called filter_nonwords that takes a list of tokens as input and returns a new list of tokens that excludes anything that isn’t entirely alphabetic. Use the str.isalpha() method to determine is a token is comprised of only alphabetic characters.

>>> text = '"Oh no, no," said the little Fly, "to ask me is in vain."'
>>> tokens = tokenize(text, do_lower=True)
>>> filter_nontokens(tokens)
['oh', 'no', 'no', 'said', 'the', 'little', 'fly', 'to', 'ask', 'me',
 'is', 'in', 'vain']

Use this function to list the top 5 most frequent tokens in carroll-alice.txt. Confirm that you get the following before moving on:

the      1642
and       872
to        729
a         632
it        595

Part 2d: Further Corpus Exploration

Iterate through all of the files in the gutenberg data directory and print out the top 5 tokens for each. To get a list of all the files in a directory, use the os.listdir function:

import os
directory = '/courses/cs159/data/gutenberg/'
files = os.listdir(directory)
infile = open(os.path.join(directory, files[0]), 'r', encoding='latin1')

This example also uses the function os.path.join that you might want to read about.

Note about encodings: This open function above uses the optional encoding argument to tell Python that the source file is encoded as latin1. Be sure to use this encoding flag to read the files in the Gutenberg corpus, as the default (Unicode) won't work!

Analysis Question #2

Loop through all the files in the gutenberg data directory that end in .txt. Is 'the' always the most common token? If not, what are some other tokens that show up as the most frequent token (and in which documents)? What do you notice about these tokens?

Analysis Question #3

If you don’t lowercase all the tokens before you count them, how does this result change, if at all? Discuss what you observe.

(When logged in, completion status appears here.)