Lab 5: Information Retrieval and Vector Semantics

We learned in class that one neat thing about SVD is that it doesn't just transform the document vectors into "concept space"; it also creates vectors for each word in the vocabulary, in the same "concept space"! This allows us to compare how related two words are (for example, in the lecture we said that "bus" and "train" would be similar since they would both point roughly in the direction of the "transit-ness" dimension).

We have provided another script, wiki_word_lookup.py, to help you play with these word vectors. If you examine the code, you should find that it looks nearly identical to the wiki_search.py script. This is by design; fundamentally the two scripts are doing the same thing (computing cosine similarities of vectors), it's just that the vectors involved mean different things. Like wiki_search.py, wiki_word_lookup.py is nearly complete, you simply need to fill in the "holes" (which, roughly, are the same as the "holes" in wiki_search.py):

TASK: Create a WikiWordVectors subclass that represents words using SVD

First, you need to implement the word vector model that WikiWordLookup will use. To this end, you will create a subclass of WikiWordVectors that you should name SVDWordVectors. Remember: we are running the same SVD as we did in the original document search! Therefore, you can start by copying the relevant SVD code from your SVDDocumentVectors. Note however that the WikiWordVectors interface is slightly different; in addition to train, you will have to implement:

• get_word_index: given a word (as a string), this function should return the index of that word in the vocabulary that was learned by TfidfVectorizer. This is necessary so we know which row of the \( V \) matrix to look up to find the vector for a given word.
• get_vocabulary: this function should return the vocabulary that was learned by TfidfVectorizer, as a list. We can later use this to do the inverse of get_word_index: given an index, we can find what the word at that index is.
• get_word_vector: this is the equivalent of get_document_vector, but because we only know about words that are in our vocabulary, we don't need to generate a new vector; given a word, we simply need to look up its row in the \( V \) matrix. You can use get_word_index as a helper function here!

TASK: Finish the implementation of WikiWordLookup

Just like in Part 2's WikiSearchEngine, WikiWordLookup is just missing the bit of code that computes the cosine similarities. Your job is to fill in that missing bit. A major hint here is to remember that wiki_word_lookup.py is fundamentally doing the same thing as wiki_search.py; cosine similarity is just math that doesn't care about what the vectors "mean". Therefore, you should not need to write any new code here; the code should be identical to its counterpart from WikiSearchEngine!

TASK: Use your SVDWordVectors subclass in main

Finally, there is one more TODO in this file, which is found in main. After parsing the command line arguments, main is supposed to initialize a vector model (i.e., a subclass of WikiWordVectors) as word_vector_model, and pass the model to WikiWordLookup. As indicated by the TODO, you simply need to fill in the code that initializes the model. Right now, we only have one subclass of WikiWordVectors, which is the SVDWordVectors that you implemented above. So, that's what you should initialize word_vector_model as.

Once you have finished all the coding tasks in this part, you should be able to successfully search for words that are similar to a given word! Answer the following analysis questions:

Analysis Question #5

Run wiki_word_lookup.py on at least 2 of the following words:

• "college"
• "frozen"
• "science"
• "programming"
• "transit"

In addition, run it on 3 more words of your choice. For all lookups, paste the results in your journal. What do you think of the results? Are any of them surprising?

Analysis Question #6

Try running wiki_word_lookup.py on the word "male". Then, run it on the word "female". Compare the results: what stands out to you? Is there anything you find troubling, or otherwise unsatisfying, about how the "female" results look compared to the "male" results?

Do the same comparison and analysis for the following pairs of words: "black" vs "white", and "africa" vs "europe".

(When logged in, completion status appears here.)

Previous Page Next Page