Lab 6: Text Generation using RNNs

Updated 4/7/21: 5:03 PM


In this lab, you'll train a language model on a corpus of text. Then, you'll use the trained model to generate text based on a primed text string. For example, you'll train a language model on Shakespeare's plays, and then, given a prompt like:
What is the temptation
you'll generate follow-on text that might look like:


You'll find useful information in chapter 10 and 11 of the fastai book.

The two corpora you'll be using are:


Part 1: Shakespeare using word tokenization


Follow the steps in Chapter 10, fairly closely:
  1. Create a language model data loader (see Language Module Using DataBlock section of Chapter 10).
  2. Show the batch and ensure it looks proper (same section).
  3. Create a non-pretrained LanguageModelLearner (see Fine-Tunting the Language Model section of Chapter 10).
  4. Train the model for one or two epochs.
  5. Generate some fake Shakespeare (at least two examples of at least 50 words, starting with the text prompt "What is the temptation" (see Text Generation section of Chapter 10).


Part 2: Shakespeare using character tokenization

In this part, you'll switch from feeding in words and predicting subsequent words to instead feeding in characters and predicting subsequent characters. You'll want to read about tokenization in the following sections of Chapter 10: Tokenization through Numericalization with fastai You'll need to: Otherwise, the steps will be fairly similar to Part 1.

Part 3: Linux source code using character tokenization

In this section you'll:
  1. Train on linux source code rather than shakespeare.
  2. Write your own greedy_predict routine that'll:
  3. Write your own random_predict routine that'll:
  4. Write your own beam_predict routine that'll:


Challenge 1 We've looked at word tokenization, sub-word tokenization, and character tokenization. Let's take it to the extreme and do bit tokenization.

Train a language model on the linux kernel corpus using bit tokenization and generate output using beam search.

Challenge 2 Rewrite random_predict and beam_predict so that they can take multiple prompts. Put the multiple prompts into a batch and calculate the predictions in parallel. For the beam prediction, do all the prediction in parallel using the batching mechanism.

This completes the lab. Submit instructions

  1. Make sure that the output of all cells is up-to-date.
  2. Rename your notebook:
    1. Click on notebook name at the top of the window.
    2. Rename to "CS152Sp21Lab6 FirstName1/FirstName2" (using the correct lab number, along with your two first names). I need this naming so I can easily navigate through the large number of shared docs I will have by the end of the semester.
  3. Choose File/Save
  4. Share your notebook with me:
    1. Click on the Share button at the top-right of your notebook.
    2. Enter as the email address.
    3. Click the pencil icon and select Can comment.
    4. Click on Done.
  5. Enter the URL of your colab notebook in this submittal form. Do not copy the URL from the address bar (which may contain an authuser parameter and which I will not be able to open). Instead, click Share and Copy link to obtain the correct link. Enter your names in alphabetical order.
  6. At this point, you and I will go back and forth until the lab is approved.
    1. I will provide inline comments as I evaluate the submission (Google should notify you of these comments via email).
    2. You will then need to address those comments. Please do not resolve or delete the comments. I will use them as a record of our conversation. You can respond to them ("Fixed" perhaps).
    3. Once you have addressed all the comments in this round, fill out the submittal form again.
    4. Once I am completely satisifed with your lab, I will add a LGTM (Looks Good to Me) comment
    5. At that point, setup an office hour appointment with me. Ill meet with you and your partner and we'll have a short discussiona about the lab. Both of you should be able to answer questions about any part of the lab.