Lab 2: Frequency and N-Grams

In this week's lecture and online lesson, we learned about the problem of sparsity, or how to handle 0 counts in language modeling. If you were to train even a bag-of-words (unigram) language model on the fiction category of the Brown corpus and then try to calculate the probability of generating the editorial category, you’d end with a 0 probability: some words that occur in the editorial category simply don’t appear in the fiction category (e.g. “badge”).

In this part of the assignment, we’ll explore that problem in more depth. At the same time, we'll get some practice working with data that is stored in formats more complicated than simple .txt.

You should put your code for this part in ngrams.py. Add the same lines to the top of this file for working with spaCy that you have in your zipf.py.

Part 2a: Extracting Data From XML Files

In Lab 1, as well as in Part 1 of this lab, we provided you with data in the form of plain-text .txt files. Such files are easy to work with, because you can just read their contents directly as a string. But plain text also has a major downside: while it can easily represent the raw text, it can't easily include metadata: that is, additional information about the data. Metadata is very important for NLP applications; we usually don't just care about the text itself, but also about properties such as who wrote it and when, or domain-specific information like "is this text toxic".

Therefore, the vast majority of actual NLP datasets (like ones you will perhaps use for your final project) are stored in more complex formats; popular options include CSV, JSON, and XML. To make sure you get experience with these common formats, from this point forward the lab assignments will have you work with this type of data instead of plain-text files.

We'll start this week with XML (and see other formats in future labs). If you're not used to XML, it's a general-purpose markup language that looks syntactically similar to HTML. Core concepts of XML that you need to know for this lab (and future ones) include:

• Nodes: these can be used to represent individual data entries. To draw an analogy to HTML: the HTML code <a href="https://cs.hmc.edu"> defines an anchor (link) node.
• Attributes: these are additional entries in a node, kind of like keys in a dictionary, that can be used to identify metadata. In the above HTML example, href is an attribute that signals that you are about to see a URL.
• Values: if each attribute is like a dictionary key, that key must also map to a specific value. In the above HTML example, the href attribute has the value "https://cs.hmc.edu".

Specifically, for this lab and Lab 3, we're going to do some work on the "Don't Patronize Me!" dataset, in which data annotators have categorized passages of text based on whether they are "patronizing or condescending towards vulnerable communities" in the hopes of supporting analysis of unconscious bias. The dataset was part of a shared task (a sort of research mini-competition) at SemEval2022.

The data for this task is in /courses/cs159/data/patronize. There’s a single XML file that contains all of the labeled examples we’ll look at this week. Each labeled example has a condescension attribute that indicates whether it was annotated as condescending: true or false.

While this dataset is small, other datasets later this semester (including those you might be interested in for your final project) can be many gigabytes. Loading an XML file that big into memory is a recipe for trouble, though, so it’s best not to store the whole thing in memory at once if we can help it. Fortunately, the lxml library gives us a way to iteratively parse through an XML file, dealing with one node at a time. Here’s sample code that opens a file called myfile.xml and call a function called my_func on every example node:

from lxml import etree

fp = open("myfile.xml", "br")
for event, element in etree.iterparse(fp, tag=("example",)):
    my_func(element)
    element.clear()

The starter code has a generator function called do_xml_parse() that uses lxml.etree to yield one node at a time. Look at that code and make sure you can explain how each line of it works before you move on.

Write a function called get_unigrams that takes as input a spaCy Doc(ument), and returns a list of all of the unigrams in the document. Like in Lab 1, get_unigrams should also take an optional argument do_lower whose default value is True. That argument should determine whether the text of each token is lowercased before returning the unigrams. For all of the analysis in this lab, you SHOULD lowercase the tokens unless told otherwise.

Next, write a function called get_examples(args, attribute, value) that returns a Counter. get_examples will use do_xml_parse to iterate through all of the examples passed in via args.examples. For each example whose attribute attribute has the value value, it will call get_unigrams on the text of the example. For example, get_examples(args, 'condescension', 'true') will call get_unigrams once for every example whose condescension attribute is true.

Hint: If you have an example element called example that contains only one piece of raw text (which is the case for all example elements in this dataset), you can access that text using example.text.

Hint: Some of the articles contain HTML entities that have been "escaped," or marked with special characters, to avoid interfering with the XML parsing. (Ampersands, or '&'s, are the biggest example of these.) You can un-escape those by doing import html and then calling the html.unescape function on the articles’ text before creating your spaCy Docs.

Stop now and confirm that if you call get_examples on the patronize_sample.xml file for the case where condescension is set to true, you get the following counts:

the: 2638
opportunity: 22
zero: 1

We are interested in knowing how many of the unigrams in one category of text (e.g., condescension='true') are not in another category of text (e.g., condescension='false'). Over the course of this assignment, you will explore several ways of grouping the text, so we’ll want to carefully organize our code for reusability. In the rest of this writeup, we’ll refer to the set of data we generate counts from as the training set, and the set of data that we check for zeros using those counts as the test set.

Write a function called compare(train_counter, test_counter, unique=False). This function will check for words in the test set that don't show up (i.e., have a zero count) in the training set. Moreover, this function can be configured to either count unique words (i.e., types) or word instances (i.e., tokens). Specifically, the three arguments to compare should be:

• train_counter: A Counter object representing counts from the training set
• test_counter: A Counter object representing counts from the test set
• unique: A boolean indicating whether to count tokens (unique=False) that don't show up in the training set or the distinct types (unique=True)

compare should return two numbers in a tuple: - The count of tokens (or types) in the test set that have a zero count in the training set - The total number of tokens (or types) in the test set

Confirm that if you call compare(Counter(['a','b','c']), Counter(['c','d','d']), unique=True) you get (1,2), and if you call compare(Counter(['a','b','c']), Counter(['c','d','d']), unique=False) you get (2,3).

The given code has a function called do_experiment that calls get_examples twice (once for the training data, once for the test data), and then prints the results from compare as a Markdown table. Read through that function now and make sure that you understand it, since you will add to it later in the lab.

Analysis Question #6

What percentage of the tokens that appear in the condescending (true) examples don’t appear in the neutral (false) examples? Conversely, what percentage of tokens that appear in the neutral (false) examples don’t appear in the condescending (true) examples? How might you initially interpret these findings—could they say something interesting about condescension?

Analysis Question #7

What happens if you look at types instead of tokens? Does this change your interpretation of the results in any way?

Part 2b: Beyond Unigrams

What happens when you move to higher order n-gram models like bigrams and trigrams?

Write a function called get_bigrams that takes as input a spacy Document, and returns a list of all of the bigrams in the document. Remember: bigrams are consecuitive sequences of two tokens, like "Los Angeles" or "my cat".

Then, write a function called get_trigrams that takes as input a spacy Document, and returns a list of all of the trigrams in the document. Remember: trigrams are consecutive sequences of three tokens, like "Harvey Mudd College" or "my cat is".

Hint: Don’t try to manually generate the bigrams and trigrams from scratch. Instead, use your get_unigrams function along with Python’s built-in zip function to save yourself some code-writing!

Modify your get_examples function so that it returns a tuple with 3 items: a Counter of unigrams, a Counter of bigrams, and a Counter of trigrams. Then modify do_experiment so that it generates three table rows with statistics for not only unigram zeros, but also bigram and trigram zeros.

Analysis Question #8

What percentage of the bigrams (tokens, not types) that appear in the condescending-labeled examples don’t appear in the neutral-labeled examples? What percentage of the bigrams that appear in the neutral-labeled examples don’t appear in the condescending-labeled examples? Then, answer those two questions again but for trigrams. Do you notice any interesting patterns/trends as you go from unigrams to bigrams to trigrams? Do the results surprise you? In your answer, make sure to include the table generated by do_experiment.

Part 2c: Randomized Chunks

Perhaps in the previous parts, you noticed interesting differences that you thought could signal differences between condescending and neutral text. But are those differences actually meaningful? In science, a fundamental question we must always address is: are the trends we are seeing just the result of random chance?

Here's one way we could address that question: instead of finding which features distinguish one category (condescending) from another (neutral), suppose we randomly break each of the categories in half. Then we could compare the feature distributions from our model using half of the condescending-labeled examples and half of the neutral-labeled examples as if they're from the same category. This can help us calibrate our sense of how much of the effect we saw in the last part is actually connected to the labels (instead of just a natural property of having lots of text).

This relates to a concept called a permutation test from statistics, a convenient way to reason about what a statistically significant result is when you don't have a clear indication of which probability distribution describes the variable you're interested in.

Randomly splitting data this way can also be useful when testing systems that try to predict something about each example using a concept called k-fold cross validation. In this scenario, you split your data into \( k \) chunks, each of which takes a turn being the test set while the other \( k - 1 \) are the training set.

In the sample data you used above, each of the examples has an attribute randomchunk that assigns it to either chunk a or chunk b. (since there are two chunks, this gives us 2-fold cross validation)

You shouldn’t need to write much (any!) code here. Instead of calling do_experiment for the condescending attribute, you can now call it with the randomchunk attribute.

Analysis Question #9

Try training on randomchunk=="a" and testing on randomchunk=="b". Then train on randomchunk=="b" and test on randomchunk=="a". How are your results different from the previous question? Why? Does this change your previous interpretations at all? Why or why not? Your writeup should include a table of your results, which you can generate with your expanded do_experiment function from above. Report percentages, not raw counts.

(When logged in, completion status appears here.)