Motivation: The Problem of Unknown Tokens
Let's suppose we have built a rule-based tokenizer following the general procedure described in the lecture: we used our expert knowledge of the English language to write a bunch of bespoke regular expressions and if-else logic, tested the logic on some dataset of real-world English text, and used the result of the test run to (1) fix any remaining edge cases, and (2) build a vocabulary. The resulting tokenizer uses our finalized regexes and logic to do an initial split of input text, discards any tokens that aren't found in the vocabulary, and returns the remaining tokens as its final output.
Now suppose we are using this tokenizer to analyze financial documents in real time. It has worked fine for the past few weeks. Today, however, a newly founded startup called WartCo has announced its intent to go public, and our system is being used to analyze WartCo's documents. Keep in mind that WartCo is a brand new company and the name "WartCo" has never appeared in any previous documents...
In the scenario as described, "WartCo" is a brand-new string we wouldn't have encountered when building our vocabulary. Thus, when the system encounters it, it will be considered out-of-vocabulary and will be treated like any other unrecognized string: discarded.
In some situations, this may be okay! If, say, your end goal is to evaluate WartCo's financial strategy, the exact name of the company doesn't matter much and it's probably safe to ignore it. On the other hand, if you really wanted to do something with the text "WartCo" other than discarding it, there are two standard approaches that could be used without having to dramatically change the logic of the rule-based tokenizer.
Option 1: Just Don't Have a Vocabulary??
Arguably, the problem with "WartCo" only arose because of the last step where we check all candidate tokens against our vocabulary. If we just skipped that step—that is, just treated all strings from the initial split as valid tokens—then "WartCo" would remain untouched!
As discussed in class, instead of looking at this as having no vocabulary, you could think of it as implicitly defining the vocabulary as "the set of all possible strings our rule-based logic could ever possibly produce".
This can be a feasible solution for simple, small-scale cases. In fact, it's exactly what you'll be doing in Lab 1—when you don't have that much data, and your rules/regexes are reasonable (e.g., "split by whitespace"), it might not always be worth it to come up with a list of "valid" tokens to use as a vocabulary. In such cases, we're trusting that the base set of rules is "good enough" and that there's nothing particularly unusual in the data.
The problem with this approach is that, well, you're keeping literally everything. That means every minor typo, every time "Massachusetts" gets misspelled as "Massachusets", every accidental random keyboard-smash, etc...those are all being kept in your final set of tokens! You can easily imagine that as your dataset gets larger, especially if you're working with messy real-world data, the amount of "debris"—random tokens that appear as the result of typos and errors—will grow out of hand.
You'll actually explore this yourself a bit in Lab 1, so we won't speak much more about it here.
Option 2: Special Tokens
In practice, developers of NLP systems typically don't want to let the vocabulary grow unbounded. Instead, they'll typically have some maximum vocabulary size in mind. As long as we have a finite-size vocabulary, there will always be the possibility of out-of-vocabulary tokens like "WartCo" appearing. But just discarding these out-of-vocabulary tokens could cause problems: for instance, if you want to count the number of words in a document, the fact that some of those words might be out-of-vocabulary tokens that get discarded could cause your counts to be off. In the most extreme scenario, a document containing only out-of-vocabulary tokens would be treated as having zero words!
One popular strategy for addressing this is: instead of discarding out-of-vocabulary tokens, we replace them with a generic token. Common choices for the generic replacement include "<OOV>" (an acronym for "out-of-vocabulary") and "<UNK>". (The angle brackets are customary to visually distinguish these from "ordinary" tokens, but of course you can choose whatever string you want).
Hay! This feels super sketchy!
-
What makes you say that?
-
Well,
"<OOV>"and"<UNK>"aren't actual words! You just made them up! 
(Sound of shuffling pages) To quote Thor from Avengers: Infinity War: "All words are made up."
It's useful to remind ourselves here that tokens and words aren't the same thing. "<OOV>" isn't a "real word" in the sense that it almost certainly never appears in real documents (well, other than in NLP texts talking about tokenization, which is actually a real concern!). But that's exactly the point: we've chosen something that is unlikely to be confused for any of the other tokens in our vocabulary, representing the "real words".
Remember that the purpose of tokenization is not to produce a perfect, lossless representation of the original text—if it was, then common steps like discarding punctuation would also be problems. Instead, the goal is to produce a "good enough" representation of the text that is practically useful in NLP applications. This "good enough" representation may diverge from the original in various ways, such as lacking punctuation or having the occassional "<OOV>" token.
-
That being said, there is still something about this approach we can find unsatisfying...
But is it Really Unknown?
Everything we've discussed so far presumes that "WartCo" is just some completely unknown token. That may be true in a mathematical sense, in that "WartCo" is not in the set $$V$$ that represents our vocabulary. But as a human, you (probably) don't just see "WartCo" as a meaningless string of characters. Even if you've never seen this string in your life, you probably see familiar patterns in it: "Co" is stereotypically a part of company names, you might associate "Wart" with Harvey Mudd, etc. This makes it hardly "unknown" in the colloquial sense!
In fact, this isn't just true for humans—go ahead and try asking any modern AI chatbot something like "What does the name 'WartCo' sound like?"
-
HMC students have free access to Gemini, in case you don't know what chatbot to use.
Specific answers may vary, but when we instructors tried it, the AI chatbot was able to successfully identify "Co" as being associated with a company name (alongside some other commentary about "Wart" being an unfortunate connotation for a company, which is totally Wally The Wart slander!). This implies that it is able to process "WartCo" as a token (or multiple tokens) somehow...but how?
Meh. It seems clear to me; "WartCo" is getting tokenized as "Wart" and "Co".
That's what I said all along, but earlier it said that was the wrong answer!
-
Yes, tokenizing it as "Wart" and "Co" seems like a plausible account of what's happening...but again, how does it know to do that?
The answer is a concept known as subwords!
(When logged in, completion status appears here.)