Key Points

• While larger context sizes would theoretically lead to better language models, in practice the fact that data is finite means that extremely large context sizes are impractical under the Markov approach we've been using.
• To make large context sizes practical, we need a way to cleanly handle cases of never-before-seen contexts, which would have probability 0 under the Markov model.
  • One solution is smoothing, which pretends that all possible contexts have been seen \( k \) more times than they were actually seen.
  • Another solution is interpolation, which combines the results of multiple language models with consecutively smaller context sizes.
  • Finally, backoff is a simplification of interpolation that also considers multiple language models, but only uses the result of one, successively reducing the context size until it reaches a nonzero probability.
• Language models are often trained with special <SOS> and <EOS> tokens that mark the start and end of sentences.
  • This allows the model to understand what tokens are more likely to start a sentence, and to know when to stop generating text.
  • Similar special tokens, such as <user>, are also what turn LLMs into chatbots.

Takeaway Skills

On a subsequent lab assignment or exam, you are expected to be able to:

• Define the bolded key terms above.
• Given some input tokens, compute add-\( k \) smoothed probabilities for some value of \( k \).
• Given some input tokens, compute consecutive \( n \)-gram probabilities of a specific token for consecutively decreasing context sizes, then apply interpolation or backoff to compute a final token probability.
• Describe how special tokens are used in modern LLM-based chatbots.

(When logged in, completion status appears here.)

Previous Page Next Page