CS 159

Recap: The Limitations of Naive Bayes

Let's start by reminding ourselves of something that was briefly touched upon in the lecture: what is "naive" about Naive Bayes?

"Naive" in this case refers to a naive assumption that forms the core of the Naive Bayes model: the assumption that all tokens are completely independent. Last week, we saw how a similar assumption in the context of generative language models results in a model that can't produce any realistic text (even if it is useful for other purposes). How does this independence assumption negatively impact the Naive Bayes model? The answer lies in the training of the model; that is, the process of learning the probabilities.

To see what kinds of problems can arise, let's walk through an example scenario.

Suppose we are training a Naive Bayes model to classify restaurant reviews as having positive (POS) or negative (NEG) sentiment. After training on a labeled corpus of restaurant reviews, we find that \( P(\text{"delicious"}|\text{POS}) \) is extremely high (close to 1.0).

Now suppose that, by some personality quirk of the users who wrote the reviews, the word "delicious" almost always occurs alongside the word "tasty", and the two words almost never appear separately. In other words, "delicious" and "tasty" are highly correlated. Based on this observation and what you know about Naive Bayes training, what do you think will be true about \( P(\text{"tasty"}|\text{POS}) \)?

Remember that in Naive Bayes, we estimate the probabilities similarly to how we estimated probabilities in our bag-of-words model last week: by taking token counts. So, \( P(\text{"delicious"}|\text{POS}) \) is estimated by counting how often "delicious" appears in documents labeled POS. But if "tasty" and "delicious" almost always occur together (i.e., in the same documents) then the count for "tasty" should be almost the same as the count for "delicious", and the two probabilities end up being nearly identical.

The reason this is a problem is that, in the Naive Bayes model, we multiply all these probabilities together to obtain our final score \( P(\text{POS}|d) \) (i.e., our confidence that \( d \) is a POS (positive) review). And so in this case, what we would be saying is that if we saw "tasty" in document \( d \) then that should offer a high contribution to the final score for POS, and that separately, if we also see "delicious" in \( d \), then that should additionally offer another high contribution to the final score.

But in some sense, if we do this, then we are double counting! We know that "tasty" and "delicious" are highly correlated. If you tell me "document \( d \) contains the word 'tasty'" then, as a human, that should give me information about what \( d \) is like and I should adjust my prediction accordingly...but if you then tell me "\( d \) also contains the word 'delicious'" then I shouldn't be surprised. I could have already guessed that \( d \) contains "delicious" the moment you told me it contains "tasty". And so this extra information shouldn't further affect my prediction, because it's not telling me anything I didn't know already.

In the language of probability and statistics: the limitation of Naive Bayes is that it is vulnerable to correlations between features in the training data. Thankfully, there are other linear models besides Naive Bayes that can address this limitation.

  • LParrot speaking

    But to understand those models, we'll first need to take a biological detour...

(When logged in, completion status appears here.)