Lab 1: Tokenization and Segmentation

Put all of your code for this part in tokenizer.py. The starter code has an empty main function that you should fill in with code to demonstrate how your functions in this part work.

You should call your main function by placing this standard Pythonic pattern at the bottom of the file:

if __name__ == '__main__':
    main()

This pattern will be useful for you as you develop more complex python programs. It allows you to write functions that will can be imported into other programs while still having your tokenizer.py be runnable as a stand-alone program. If you want to run Python (or ipython) in interactive mode with a particular file, use the -i flag: python -i filename.py (or ipython -i filename.py if you prefer ipython). This will let you load the code in filename.py and then run those functions.

The main function should be the only place where you print anything.

For this part of the assignment, as well as Part 2, we provide a minimal test suite that you can use to double-check that your functions are working as expected. You can run the tests as python test-tokenizer.py. Again, these are minimal tests and their main purpose is to help you double-check that you're interpreting the instructions correctly, not to catch every possible bug! You are welcome to add your own tests—and if you do, that would be something to mention in your journal! (As a reminder, the journal is the only thing that is actually graded; the output of these tests is never directly factored into your grade for this assignment)

Hint: every function in Part 1 is possible to implement in 1-2 short lines of code using Python standard classes! If you find yourself writing many lines of complex logic, you may want to pause and peruse some of the documentation pages we have linked throughout.

Part 1a: get_tokens

Write a function called get_tokens that takes a string s as its only argument. The function should return a list of tokens in s, in the exact order that they appeared. For the purposes of this question, we define a token to be any space-separated item. For example:

>>> get_tokens('The cat in the hat ate the rat in the vat')
['The', 'cat', 'in', 'the', 'hat', 'ate', 'the', 'rat', 'in', 'the', 'vat']

Hint: If you don’t know how to approach this problem, read about str.split().

Part 1b: count_tokens

Write a function called count_tokens that takes a list of the tokens in s as its only argument and returns a collections.Counter that maps a token to the count of how many times it occurred in s. For now, you can use the output of the get_tokens function as the input to this function (but your function should be generic such that it can be applied to any list of tokens, for example if—spoiler alert!—we later have a different tokenizer we want to test).

>>> s = 'The cat in the hat ate the rat in the vat'
>>> toks = get_tokens(s)
>>> count_tokens(toks)
Counter({'the': 3, 'in': 2, 'The': 1, 'cat': 1, 'hat': 1, 'ate': 1, 'rat': 1, 'vat': 1})

Notice that this is somewhat unsatisfying because "the" is counted separately from "The". To fix this, have your get_tokens function be able to lower-case all of the tokens before returning them. You won’t want to break any previous code you wrote, though (backwards compatibility is important!), so add a new parameter to get_tokens with a default value:

def get_tokens(s, do_lower=False)

Now, if get_tokens is called the way we were using it above, nothing will change. But if we call get_tokens(s, do_lower=True) then get_tokens should lowercase the string before getting the words. You can make use of str.lower to modify the string. When you’re done, the following should work:

>>> s = 'The cat in the hat ate the rat in the vat'
>>> toks = get_tokens(s, do_lower=True)
>>> count_tokens(toks)
Counter({'the': 4, 'in': 2, 'cat': 1, 'hat': 1, 'ate': 1, 'rat': 1, 'vat': 1})

Part 1c: tokens_by_frequency

Write a function called tokens_by_frequency that takes a list of tokens as its only required argument. The function should return a list of (token, count) tuples sorted by count such that the first item in the list is the most frequent item. Items with the same frequency should be in the same order they appear in the original list of tokens.

tokens_by_frequency should, additionally, take a second parameter n that specifies the maximum number of results to return. If n is passed, then only the n most frequent tokens should be returned. If n is not passed, then all tokens should be returned in order of frequency.

>>> tokens_by_frequency(tokens)
[('the', 4), ('in', 2), ('cat', 1), ('hat', 1), ('ate', 1), ('rat', 1), ('vat', 1)]

>>> tokens_by_frequency(tokens, n=3)
[('the', 4), ('in', 2), ('cat', 1)]

(When logged in, completion status appears here.)