Comparing Smoothing, Interpolation, and Backoff
Once again, we've got a whole buffet of options! I love it!
...But which one should I actually use?
Each option has its pros and cons:
- Smoothing is the simplest to implement, since it doesn't require training models and can be implemented as just a single (relatively) straightforward mathematical calculation. However, it does not directly capture our human intuition about sometimes not needing the whole context, and instead just blindly assigns extra probability to every possible token sequence.
- Backoff is more complicated than smoothing, but it is the simplest way to capture the human intuition that sometimes only part of the context is needed.
- Interpolation is arguably more complete than backoff since it actually combines multiple models rather than just choosing between them, but it requires extra training to learn per-model weights.
To be quite frank, these days very few people bother optimizing their Markov language models in any real way. Interpolation, in particular, is a bit of a relic from the days when Markov language models were the only viable language models. RNNs marked the beginning of neural network based language models (which these techniques don't apply to), and transformer-based LLMs were the nail in the coffin. Today, Markov language models are only really used as simple demos or baselines, and in that context, optimization doesn't really matter. Therefore, smoothing tends to be the most commonly used approach these days for handling zero probabilities, since it is easy to implement. Backoff is occasionally used as well, while interpolation is rarer.
There are still niche research areas where Markov language models, and interpolation, might still be relevant though!
And there is nothing wrong with working in niche areas...after all, the late HMC Prof Robert Keller's work on neural networks was once considered niche, and well, look at us now!
This may be a good time to take a stretch break, as the next part of the lesson will be pivoting to a completely different topic.
(When logged in, completion status appears here.)