The Softmax Function

Let's revisit the linear classifier function we have so far:

$$ \hat{\mathbf{y}} = F(\mathbf{W}\mathbf{x} + \mathbf{b}) $$

We need a choice of activation function \( F \) that will make the result \( \hat{\mathbf{y}} \) a valid probability distribution. Remember that in a valid probability distribution, all values are between 0 and 1 (inclusive) and they must sum to exactly 1.

At this point, we could go down a whole rabbit hole of mathematical derivations. But this isn't a theoretical machine learning class, so we won't do any of that and just skip to the end instead.

It turns out that the most commonly used activation function for turning a vector of arbitrary values into a valid probability distribution is the softmax function, often denoted as \( \sigma \). The softmax function has the following definition (where \( \mathbf{z} \) is the input vector, and \( z_i \) are its individual elements):

$$ \sigma_i(\mathbf{z}) = \frac{e^{z_i}}{\sum_{j=1}^k e^{z_j}} $$

In plain English, what this is saying is: softmax first takes the exponential of every element in the input vector, and then normalizes the whole vector in the typical way (dividing by the sum).

And so, our final linear classifier equation is:

$$ \hat{\mathbf{y}} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b}) $$

Training a Linear Classifier

So far, we've been proceeding under the assumption that we already had our weights \( \mathbf{W} \) and our biases \( \mathbf{b} \). But of course, in reality there is no "weights fairy" magically handing out weights! Just like in the case of Naive Bayes, we have to learn the weights and biases from data. Unlike in the case of Naive Bayes, it's unclear exactly how we do that.

So let's start by establishing some basic building blocks. We ultimately want our classifier to be as correct as possible. But how do we formalize that idea?

Remember that \( \mathbf{x} \) in the equation above is an input feature vector; in NLP, this represents some document from our corpus. Suppose that document was labeled as belong to class \( c_{t} \). In other words, the answer to the yes-or-no question "does this document belong to class \( c_{t} \)" is yes (1), and conversely, for all other classes \( c_o \), the answer to "does this document belong to class \( c_o \) is no (0).

If our classifier tends to be correct, we should expect it to assign a high probability to \( c_{t} \) and a low probability to everything else. So we want to establish a loss function that rewards assigning a high probability to \( c_t \). "Rewards" in this case means that the loss function should output a value that is close to 0, indicating "low error". Likewise, it should also reward assigning low probability to everything else.

Once again, at this point we'll have to skip the technical details, as there's many cost functions one can choose from. But we will show just one simple example to emphasize that cost functions aren't "magic". The logistic loss function can be defined for binary (1 or 0) outputs as follows:

$$ \text{cost}(\hat{\mathbf{y}}_i, \mathbf{y}_i) =
-\log \hat{\mathbf{y}}_i \text{ if } \mathbf{y}_i = 1 \text{,} $$

$$ \hspace{64px} -\log (1 - \hat{\mathbf{y}}_i) \text{ if } \mathbf{y}_i = 0 $$

Let's check that this has the desired behavior:

• The true label \( \mathbf{y}_1 \) is 1 and the predicted probability \( \hat{\mathbf{y}}_i \) is high (which is good): we take the negative log of a number that is close to 1, giving us something close to 0. ✅
• The true label \( \mathbf{y}_1 \) is 0 and the predicted probability \( \hat{\mathbf{y}}_i \) is low (which is good): we take the negative log of (1 minus a number that is close to 0), which ends up being the negative log of a number that is close to 1, giving us something close to 0. ✅
• The true label \( \mathbf{y}_1 \) is 1 but the predicted probability \( \hat{\mathbf{y}}_i \) is low (which is bad): we take the log of number that is close to 0, giving us an extremely large value ("high error"). ✅
• The true label \( \mathbf{y}_1 \) is 0 but the predicted probability \( \hat{\mathbf{y}}_i \) is high (which is bad): we take the negative log of (1 minus a number that is close to 1), which ends up being the negative log of a number that is close to 0, giving us an extremely large value ("high error"). ✅

Having a way to measure the correctness of a classifier turns out to provide the last missing ingredient we need! Suppose we randomly initialize the weights \( \mathbf{W} \) and biases \( \mathbf{b} \). Now we can actually compute \( \hat{\mathbf{y}} \) and compute the loss \( \text{cost}(\hat{\mathbf{y}}, \mathbf{y}) \). This will almost certainly have very high loss, since we used random weights and biases.

But now, notice the following: if we treat \( \mathbf{W} \) and \( \mathbf{b} \) as variables (because they are what we're trying to find values for), then \( \text{cost}(\hat{\mathbf{y}}, \mathbf{y}) \) is a function on \( \mathbf{W} \) and \( \mathbf{b} \)! And that means we can take its derivative (gradient) with respect to the variables \( \mathbf{W} \) and \( \mathbf{b} \).

We'll never ask you to actually compute a gradient in CS 159. The only thing we want you to understand is that you could do it just by applying the good old chain rule (of calculus, not probability). And if you think back to your calculus classes, you should remember that the gradient is a generalization of the concept of slope; it tells us the direction of change. Which means if we adjust \( \mathbf{W} \) and \( \mathbf{b} \) in the direction of the negative gradient, we end up reducing the value of the function \( \text{cost}(\hat{\mathbf{y}}, \mathbf{y}) \). In other words, we find a new value of \( \mathbf{W} \) and \( \mathbf{b} \) that leads to lower loss (i.e., a more "correct" classifier)! And we can keep doing this over and over until we reach a minimum (the loss stops decreasing).

This algorithm is known as gradient descent, and a linear classifier trained using softmax activation and the logistic loss function is known as a logistic regression model. We won't go over the proof of this, but it turns out that logistic regression is a generalization of Naive Bayes that doesn't assume the features are independent. This means in practice it tends to do better than Naive Bayes, and so researchers often use it as their "default" linear classifier whenever they are first setting up an experiment.

What to Know About Logistic Regression

Given that CS 159 is not a machine learning class, we don't need you to be intimately familiar with the mathematics of logistic regression, nor do we need to know how to implement it. We do need to you to know the following properties of logistic regression:

• It's a linear classifier.
• Not only that, it's one of the most commonly used linear classifiers and is usually superior to Naive Bayes.
• It uses the softmax activation function (which you should be familiar with).
• It is trained using gradient descent (which you don't need to be able to do by hand, but should be able to describe in words at a high level).

If you ever want to use logistic regression (for instance, on your final project), the scikit-learn package has a full implementation available that you can just import, so you'll never need to implement any of the math above in CS 159.

(When logged in, completion status appears here.)

Previous Page Next Page