Key Points
- The problem of unknown tokens is a major sticking point for rule-based tokenizers.
- Rule-based tokenizers are inflexible and cannot cleanly handle tokens that are not in their vocabulary.
- The most common solution is to replace out-of-vocabulary tokens with a generic token like
"<OOV>", but this loses semantic meaning.
- Humans (and LLMs) are able to process unknown tokens, in part, by breaking them up into subwords: character sequences within a word that can (but do not have to) carry semantic connotations.
- For example, "remanualism" is not a real word, but a human would probably process it as something like "re", "manual", "ism".
- Subword tokenizers are tokenizers that can split text into subwords.
- This does not inherently imply lack of rules, but in practice almost all subword tokenizers are not rule-based and instead use some kind of learning algorithm.
- Byte-Pair Encoding (BPE) is a popular subword tokenization algorithm used in GPT and other popular LLMs.
- During training, the BPE algorithm first splits the text into characters, then iteratively merges the most common token pairs, adding each merged pair to the vocabulary as a new token.
- The trained tokenizer can be used to process new text, where it works basically the same way as in training, except that it cannot grow the vocabulary and is therefore limited to doing merges that result in tokens that are already known in the vocabulary.
- Tools for evaluating tokenizers include the confusion matrix and its derived metrics: precision, recall, and F1.
- Computing these metrics requires some kind of benchmark containing labels of what is and is not a valid token.
Takeaway Skills
On a subsequent lab assignment or exam, you are expected to be able to:
- Define the bolded key terms above.
- Describe a scenario where a rule-based tokenizer would produce a result that is unsatisfying from a human perspective, but a subword tokenizer would produce more intuitive results.
- Execute the BPE algorithm by hand (training and/or tokenization on new data) on a given small dataset.
- Compute a confusion matrix based on a provided scenario of labels and predictions.
- Compute precision, recall, and F1 from a confusion matrix.
(When logged in, completion status appears here.)