Evaluating Classifiers

To wrap up our discussion of text classification (which started last week with Naive Bayes, and ends here with neural networks), we'd like to take a step back and address a question of evaluation: how do you tell if a classification model is "good"?

Generalizing the Confusion Matrix

If your classification task only has two classes, like "spam" versus "not spam", then we can pretty straightforwardly apply the concept of the confusion matrix, which we learned about in the Week 2 Online Lesson. You just need to designate one class as the "positive class", then we can fill out the true and false positives, and the true and false negatives. For example, if we treat "spam" as the positive class:

                         Classifier's Prediction
                      |    SPAM    |   NOT SPAM   |
---------------------------------------------------
Actual Label     SPAM |     TP     |      FN      |
             NOT SPAM |     FP     |      TN      |

And once we have these numbers, we can compute precision, recall, and F1 score the exact same way we did in Week 2.

If we have more than two classes, things aren't as straightforward, because there's no obvious way to pick one class to be the "positive class". We usually don't think one class is more important than the others. For instance, if we're doing authorship attribution with classes [ "Shakespeare", "Chaucer", "Keats" ], it's not as if one of the authors is the one we "actually care about"; we care about all of them!

The way to resolve this is to recognize that, in multi-class settings, we can't fully describe the performance of the model with only one number for precision, one number for recall, etc. Instead, each metric will need to be computed multiple times, one for each class.

What we'll do is that each class gets a turn at being the "positive class". When it's class \( c_i \)'s turn to be the positive class, we'll define true/false positives and true/false negatives as follows:

• A true positive means that the actual label was \( c_i \) and the classifier's prediction was \( c_i \)
• A false positive means that the actual label was anything other than \( c_i \) but the classifier's prediction was \( c_i \)
• A true negative means that the actual label was anything other than \( c_i \) and the classifier also predicted anything other than \( c_i \) (even if the predicted and true label don't match!)
• A false negative means that the actual label was \( c_i \) but the classifier's prediction was anything other than \( c_i \).

With these four basic ingredients defined, we can once again compute precision, recall, and F1 using their standard formulas. The catch is that we won't just have one number for each metric. Instead, each metric will be separately computed \( n \) times, since each of the \( n \) classes takes a turn as the "positive" class.

If you want to reduce each metric back down to one number, you can just take the average. One thing to be careful about, however, is that classes can often be imbalanced: sometimes one class may be overrepresented in the data compared to the other classes (e.g., non-spam is much more common than spam). You therefore have to make a decision about whether to just take a straight average, or compute a weighted average based on the class distribution. Taking a straight average is referred to as a macro average, while taking a weighted average is referred to as a micro average.

An alternative metric you'll often see in classification, which is less common in other settings, is accuracy. Accuracy is simply computed as "the percentage of correctly predicted examples". Mathematically,

$$ \text{Accuracy} = \frac{\text{Number of examples where the classifier's prediction is exactly equal to the true label}}{\text{Total number of examples in the data}} $$

Generalizability, train-test splits, and cross-validation

Notice that all of the metrics we've defined so far need to be computed on some actual data, with ground-truth labels indicating the correct answer for each example in the data. Remember also that every classifier we've discussed so far needs to be trained on some labeled data (for example, in Naive Bayes we must count things in the data to estimate the necessary probabilities).

Given that we already had some labeled data for training, it may be tempting to re-use that same labeled data when computing metrics like precision/recall or accuracy. But if you do this, you may not get an accurate picture of the classifier's true performance! It's possible that the classifier has learned things that are too specific to the training data, and that the classifier would therefore fail to generalize to other data it hasn't seen before. This phenomenon is known as overfitting.

In general, testing for generalization is an extremely difficult problem. After all, for any given training data, there's an absolutely huge amount of data that isn't in the training data (literally, everything else in the world!). There's no way we can realistically evaluate our model on all the other data out there...and even if we could, there is still data that doesn't exist yet because it will be created in the future.

So instead, researchers typically estimate generalization by doing something called a train-test split. Given some training data, they will choose some subset to hold out and not include during training. This held-out data is called the test set, while the rest is known as the training set. They can then train the model on the training set, but use the test set to compute metrics like precision and recall. Because the model never saw the test set data during training, we can be reasonably confident that good performance on the test set indicates that the model has learned something generalizable, rather than just memorizing the training data.

Of course, there's no reason to stop at just one train-test split! You can also repeat this process multiple times with different test sets. For example, consider taking your data and splitting it into three equal-size "chunks". Then you can do three rounds of training and testing:

• Round 1: train on chunks 2 and 3, test on chunk 1.
• Round 2: (starting from scratch) train on chunks 1 and 3, test on chunk 2.
• Round 3: (starting from scratch) train on chunks 1 and 2, test on chunk 3.

This is just an example; in general you can have as many chunks as you want. With \( k \) chunks you get \( k \) separate rounds of training and testing. This is known as \( k \)-fold cross validation.

(When logged in, completion status appears here.)