Lab 5: Information Retrieval and Vector Semantics

As discussed in class, one problem with using the term-document matrix for IR is that cosine similarity on those vectors depends on exact word overlap. One solution we discussed was to learn higher-level "concepts" using SVD—a rudimentary version of the larger concept of vector semantics. In this part of the lab, you'll apply SVD to the term-document matrix to transform your original document vectors into "concept space" where each column represents some higher-level concept that is correlated with multiple words. Then, you'll observe how this transformation impacts the search results.

TASK: Create a WikiDocumentVectors subclass that represents documents using SVD

Our main information retrieval system in WikiSearchEngine was designed to be generic, so thankfully as long as we simply implement an SVD vector model, we can just plug that model in to WikiSearchEngine without having to change any of the IR code. To this end, you will create a new WikiDocumentVectors subclass, which you should name SVDDocumentVectors.

Remember that SVD works by transforming a term-document matrix, so SVDDocumentVectors will share quite a bit of code with your previous CountDocumentVectors. Therefore, you should start by copying the code from your CountDocumentVectors (yes, I know copy-pasting code is considered bad practice, but we'll let it slide for this lab since otherwise we'd need to overcomplicate the inheritance structure of the classes...) . Then, you can proceed by making a few simple modifications:

1. In train, instead of directly returning the term-document matrix, you should instead run SVD on it. The matrix you get from TfidfVectorizer is a sparse matrix (which you learned about in Lab 3), so to do SVD on it, you can use the svds function from the scipy.sparse library. For the number of latent dimensions (i.e., the number of "concepts" we want to learn), you should set the k parameter to 100. svds will return three matrices: U, s, and Vh. These correspond to \( U \), \( \Sigma \), and \( V^T \) from the lecture. As a reminder, \( U \) contains the document vectors in "concept space" (so, this is what you want to return), \( \Sigma \) contains the singular values (which, for our purposes, we'll ignore), and \( V^T \) contains the word vectors in "concept space" (but transposed so the words are on the columns). You will want to hang on to \( V \) somehow, as we'll need it later!
2. In get_document_vector, after computing the query vector using TfidfVectorizer, you need to similarly transform that vector to "concept space". As a reminder, this can be done by a matrix multiplication with \( V \) (which, as noted above, you should have somehow held on to after training!). As usual, be careful about the matrix shapes when doing matrix multiplication! Intuitively, the result we want is a single vector representing the query in "concept space". So, this should be a single-row 1xk matrix, where k is the number of latent dimensions (which we previously set to 100). I recommend printing the matrix shapes to make sure you're getting them right; don't hesitate to reshape and/or transpose as needed!

TASK: Add a new command-line argument to enable the use of SVD vectors

Now that we've defined our SVD vector model, we need to modify the main function to let us specify that we want to use the SVD vectors in WikiSearchEngine. Make the following modifications to main:

1. Add a new argument --model / -m to the argument parser. This argument should be a string that can take two possible values: "tfidf" or "svd", and the default value should be "tfidf".
2. Modify the initialization of the doc_vector_model variable, which in Part 2 you initialized as a CountDocumentVectors object. You should add an if statement so that if args.model is "svd", then doc_vector_model instead gets initialized as a SVDDocumentVectors object. If args.model is "tfidf" (the default), then doc_vector_model should remain as a CountDocumentVectors object.

Once you have made these changes, then running:

python wiki_search.py /courses/cs159/data/wikipedia/enwiki_sampled/ -q "liberal arts college in southern california" -m svd

should run the same "liberal arts college in southern california" search from before, but using SVD vectors.

Use this setting to answer the following analysis questions:

Analysis Question #3

Re-run the same queries from Analysis Question #1 using your new SVD vector model, and paste the new results in your journal. What changed? For each query, do you think the new results are better or worse than before? Why?

Analysis Question #4

Re-run your queries from Analysis Question #2 using your new SVD vector model, and paste the new results in your journal. What changed? Do the new results better align with your expectations? For the two queries you originally expected the IR system to get wrong, does the system now get them right?

(When logged in, completion status appears here.)