Subword Tokenization
What is a Subword?
So far, we've been talking about words being represented as strings. The notion that strings themselves are made up of characters hasn't really gotten much focus past the first lecture. In practice, however, humans don't interpret words based only on "do I recognize this exact string?". We assign meaning to the sequences of characters within the word.
This might become more clear if you try it yourself...
Most people would probably say that "remanualism" looks more like a plausible English word than the other candidates. And this is because English words may be sequences of characters, but they are not random sequences of characters; there are structures and patterns to them. In this case, we might recognize the following patterns in "remanualism":
- "re" is a prefix that indicates doing something again or multiple times;
- "manual" is an actual word meaning to do something by hands or without tools;
- "ism" is a suffix that usually indicates a belief system or philosophy
As we can see above, these patterns also have meanings associated with them! From this breakdown, we might speculate that if "remanualism" was a real word, it might mean something like "the philosophy that society should return to doing things by hand".
BRB, gonna register "remanualism.org" real quick.
"re" and "ism" are not words on their own. They are examples of subwords: character sequences that are commonly seen within words. And while not all subwords have meanings associated with them (for example, "th" is common in English words but doesn't mean anything on its own), many do.
-
In English, many meaning-laden subwords are the result of the language borrowing quite liberally from other languages like Greek and Latin.
Examples of this include "aero" being associated with aviation, "cyber" being associated with computers, "meta" meaning above or beyond, etc.
Subword Tokenization
With all of that in mind, what is subword tokenization?
Straightforwardly put, a subword tokenizer is any tokenizer that can produce tokens that are chunks of words—in other words, that can produce subwords. For example, a tokenizer that splits "MuddCo" into "Mudd" and "Co" would count as a subword tokenizer, since "co" is a subword.
-
Pretty much every modern LLM uses subword tokenization, which is what allows them to infer the meaning of "made-up words" like "MuddCo".
You may notice that this definition doesn't say anything about whether or not the tokenizer is rule-based. Indeed, it is totally possible to write rules that split words into subwords, and some rule-based tokenizers may contain rules to handle common cases like "-ing". In practice, however, there are so many possible subwords and combinations of subwords that it is exceedingly difficult to write rules that can handle every possible case. Therefore, when we talk about subword tokenizers, we are almost always referring to tokenizers that aren't rule-based and instead follow some algorithm to automatically learn subwords from data. In this lesson, we'll cover one such algorithm: byte-pair encoding, or BPE.
(When logged in, completion status appears here.)