Lab 5: Information Retrieval and Vector Semantics
Once again, we have provided you with starter code that implements much of the information retrieval functionality already. There are TODO comments in all three files indicating "holes" in the starter code that you will need to fill in.
TASK: Comment the starter code
Just like the last two labs, your first job is to add comments throughout the provided code, including docstrings for every function. Once again, we will be reading your comments and they will contribute to your completeness and correctness scores! Therefore, you should aim to write comments that prove to us that you fully understand what the code is doing.
There is no analysis question for this part of the assignment; your comments are the main deliverable. In addition, keep in mind that if you encounter syntax/functions that are unfamiliar to you and need to do some research to figure them out, that is something you should explicitly discuss in your journal!
TASK: Working with JSON
Way back in Lab 2, we introduced you to the notion that real datasets typically aren't stored as plain text and instead are stored in structured formats, which enable metadata to be added alongside text. We showed you one common structured format, XML, which you worked with in Labs 2 and 3.
This week, we're introducing you to another commonly used format: JSON. JSON stands for JavaScript Object Notation, and as the name implies, it's meant to store data in a format that looks a lot like JavaScript objects. If you've never used JavaScript before, don't worry—a JavaScript object is basically the same thing as a Python dict. So, one way to think about JSON is that it's a way of saving dicts to a file!
The directory /courses/cs159/data/wikipedia/enwiki_sampled contains our Wikipedia dataset, split across multiple .jsonl files. Note that a .jsonl file is slightly different from a .json file. The l stands for "list" or "lines", and it indicates that the file contains multiple JSON objects (i.e., dicts), one on each line (by contrast, a regular .json file contains only one object). Each JSON object represents the full contents of one Wikipedia article.
Your coding task for this part of the assignment is to finish the currently-incomplete JSON loading code in the WikiArticles class, found in WikiDataReader.py. Specifically, you need to fill in to TODOs in the _read_wiki_file function such that the function will do the following for each line in the file fp:
- Load the line as a JSON object (i.e., dict) using Python's
jsonlibrary - Extract the
nameattribute from the object (this represents the name of the Wikipedia article) - Extract the
abstractattribute from the object (this represents the abstract, which is the summary text you see at the top of the article; this is the text we'll be searching over)
Hint: There are three TODOs in _read_wiki_file, and each one can be addressed using only one line of code. So, for each line, the only thing you should need to do is replace = None with some actual code that does what the TODO describes.
Although no tests are provided, you may want to temporarily add some print statements to make sure the data is getting loaded and read properly.
(When logged in, completion status appears here.)