Lab 3: Text Classification
In previous labs, the starter code has mostly just been a blank skeleton, and you've been asked to fill in most of the actual code. However, as we progress through the semester, we'll be covering more and more high-level concepts that depend on the more foundational concepts covered in the first few weeks (and labs). For example, pretty much every lab will depend on tokenization. But if you had to write the same tokenization function over and over again, that would be a very inefficient use of your time! Therefore, starting from Lab 3, there will be a bit more "meat" to the starter code. The starter code will come with several already-implemented or mostly-implemented functions. These functions will either implement basic concepts that you've already implemented before in the past, or act as helper functions that do stuff that is required by the code but is tangential to the main learning outcomes.
Of course, it's important to not just blindly use code that you find on the internet! Before you start any lab, you should make sure you have read the provided starter code and fully understand what it's doing.
To enforce this understanding, this week your first task will be to add comments to PCLDataReader.py, including docstrings for every function. We will be reading your comments and they will contribute to your completeness and correctness scores! Therefore, you should aim to write comments that prove to us that you fully understand what the code is doing.
The code may make use of Python features or library calls that are unfamiliar to you. This is by design—if you see syntax or a function that you haven't seen before, you should look it up. The ability to get comfortable with code that looks unfamiliar is very important for doing NLP research (and research in general)...and will also come in handy when it comes time for the final project. If you encounter things that are unfamiliar or confusing, be sure to make a note of it in your journal, and cite whatever resources you used to figure it out!
In particular, the following functions / syntax may be new to some of you, and you should take extra care to make sure your comments and journal demonstrate that you have achieved full understanding of them:
isliceyieldABC@abstractmethod- scikit-learn
DictVectorizer
After you've finished going through the starter code, use your understanding of the code to answer the following analysis questions:
Analysis Question #1
Instead of directly returning a list of examples, do_xml_parse uses the yield keyword. What does this keyword do, and why have we chosen to use it? Your answer must directly address the issue of resource usage (e.g., memory, processing) and efficiency. Your answer may (but doesn't have to) make comparisons to the function short_xml_parse, which does directly return a list of examples (and which is not actually used anywhere else in the code, because it's bad!).
Analysis Question #2
In the PCLFeatures class, we initialize a scikit-learn DictVectorizer with the parameter sparse=True. This tells DictVectorizer to store our features in a sparse matrix. Take a moment now to read up on sparse matrices. What is the importance of setting sparse=True?
Hint: your answer here should refer to very similar points you brought up in Analysis Question #1!
Note before you continue: Notice that the process methods in PCLFeatures and PCLLabels take a max_instances optional parameter. In both cases, this argument is used to help determine the size of matrices they create, and they are used as an argument to the do_xml_parse function. When you are working on debugging your code, you should pass in a value for max_instances that is small to help you iterate quickly (see the provided tests for examples). For example, when you are first starting out, you might want to set max_instances to something very small, like 5 or 10. Once you’re a bit more confident, you can set max_instances to a value that is small enough for it to run quickly but large enough that you’re confident things are working, for example 500. If you set max_instances to None, it will read through the XML file and determine the largest possible value for max_instances (which in this case is just over 10,000).
(When logged in, completion status appears here.)