Lab 5: Information Retrieval and Vector Semantics

Now that we've finished handling the data loading, it's time to dive into some information retrieval!

For this part of the lab (and Part 3), you will be working with the wiki_search.py file. This file can be run as a script on the command line with the following arguments:

usage: wiki_search.py [-h] -q Q [-n N] data_dir

positional arguments:
  data_dir              Directory containing the Wikipedia corpus files

options:
  -h, --help            show this help message and exit
  -q, --query Q         The search query to run
  -n, --num_articles N  Only keep the first N articles parsed from the corpus files

So for example, to run a search for "liberal arts college in southern california", I would run:

python wiki_search.py /courses/cs159/data/wikipedia/enwiki_sampled/ -q "liberal arts college in southern california"

Of course, if you try to run this now, it won't do anything, since we haven't yet implemented the bits that do information retrieval. Let's fix that!

Note: Notice the --num_articles parameter. When debugging, it may be a good idea to use that parameter to limit the size of the data you read in, so that the code can run faster and you can debug faster. However, when answering analysis questions, you should not use the --num_articles parameter, so that the script reads in all of the articles.

TASK: Create a WikiDocumentVectors subclass that represents documents using a term-document matrix

WikiDataReader provides an abstract base class WikiDocumentVectors, which specifies a standard interface for working with vector representations of documents. The advantage of this approach is that it will let us write generic code where we can swap out what vector representation we are using for IR.

Just so we have something to start with, let's implement the most basic type of vector representation: a term-document matrix, like the one we learned about in class. Recall that a term-document matrix is a matrix \( T \) which is indexed by documents on the rows and vocabulary words on the rows, such that \( T_{i,j} \) is the count of word \( j \) in document \( i \).

As described, it sounds like computing a term-document matrix is a lot of work: we would need to tokenize text, build a vocabulary, then compute counts for all words in all documents. That sounds...doable, but tedious! Thankfully, we can avoid the tedium, because scikit-learn provides a single class that does all that work for you: TfidfVectorizer. That means, unlike in previous labs, you won't need to think at all about tokenization or vocabulary building; TfidfVectorizer will do it all for you and then just give you the term-document matrix! Aren't libraries amazing???

At the top of wiki_search.py, you should write a subclass of WikiDocumentVectors, which you should name CountDocumentVectors. This subclass should use TfidfVectorizer to both build a term-document matrix for the Wikipedia corpus (the train function), and create vector representations for any new search queries that come in (the get_document_vector function).

When initializing a TfidfVectorizer instance in your implementation, you must use the following settings:

• Set the maximum vocabulary size (max_features) to 10,000.
• Ignore any words that appear in more than 70% of the documents. This can be achieved using the max_df parameter. This is the same thing as the stop_words argument you saw in PCLVocab in Lab 3—a way of excluding the most common words from the vocabulary so that they don't add noise to the process.

Hint: if you use TfidfVectorizer properly, every function in your new subclass (the constructor, train, and get_document_vector) should only require one line of code. This is because your new subclass basically just acts as a thin wrapper around TfidfVectorizer.

TASK: Finish the implementation of WikiSearchEngine

We have provided a near-complete implementation of an information retrieval engine in the WikiSearchEngine class. This class is a generic implementation of information retreival: you give it the vectors to use, and it does the task of computing cosine similarities to find the most similar documents to a query.

Right now, most of the code is complete but it is missing the most important part: the part that actually takes the query vector and computes its cosine similarity with all document vectors. This missing part is marked with a TODO, and your job is to fill in the missing code to compute cosine similarities. Just like in Lab 4, you should avoid using for loops. Once again, it is possible to compute all the cosine similarities at once using a single numpy operation!

TASK: Use your CountDocumentVectors subclass in main

Finally, there is one more TODO in this file, which is found in main. After parsing the command line arguments, main is supposed to initialize a vector model (i.e., a subclass of WikiDocumentVectors) as doc_vector_model, and pass the model to WikiSearchEngine. As indicated by the TODO, you simply need to fill in the code that initializes the model. Right now, we only have one subclass of WikiDocumentVectors, which is the CountDocumentVectors that you implemented above. So, that's what you should initialize doc_vector_model as.

Once you have finished all the coding tasks in this part, you should be able to successfully run queries! Answer the following analysis questions:

Analysis Question #1

Try the following queries:

• "first manned mission to the moon"
• "liberal arts college in southern california"
• "marvel movie about an insect themed superhero"

For each query, paste the results you get, and then discuss: how satisfied are you with these results? Do at least some of the top matches appear relevant to the query? Do you think that at least one of the returned articles matches the intent of the query? For results that seem clearly incorrect, briefly speculate on what is causing them to score highly for the query.

Analysis Question #2

Come up with 4 queries of your own, in the following categories:

• 1 query where you think a human would have an easy time finding the answer, and where you expect your IR system to get it right
• 1 query where you think a human would have an easy time finding the answer, but where you expect your IR system to get it wrong
• 1 query where you think a human would have a hard time finding the answer, but where you expect your IR system to get it right
• 1 query where you think a human would have a hard time finding the answer, and where you expect your IR system to also get it wrong

Briefly justify your reasoning for each query. Then, report the results of running your IR system for each query. For each query, did the IR system behave the way you expected, or were you surprised? Why?

(When logged in, completion status appears here.)