From Neurons to Networks

Perceptrons and logistic regression are not the same thing. But as they are both linear classifiers, they are quite similar—once they've been trained, they really only differ in terms of their activation function.

For the remainder of this lesson, let's use the following diagram to denote a generic "linear unit" that applies the linear operation \( \mathbf{w} \cdot \mathbf{x} + b \) and then applies an activation function \( f \):

Of course, this is just for a single output. As previously discussed, we typically want one output per class (e.g., to produce a probability distribution over classes) so in practice an actual classifier contains multiple of these "linear units", each of which independently applies some weights and biases to the input features:

Fancier Features?

Let's take a moment to think about features. We've explained that a feature is any property of the input text that we might find useful to our classification task. In lecture, most of the features we described were simple: either basic token frequencies, or low-level properties like string length, number of capital letters, etc. And the reason for this is that, so far, we've been thinking of features as things you have to manually implement. Writing Python code to compute the number of capital letters seems straightforward (or at least doable), so it's a plausible feature.

But if we limit ourselves to easily-implementable features, we quickly start to feel frustrated. Think back to the spam classification activity we did in class. Maybe you started with surface-level questions like "does it have all caps?", but chances are that many of you tried to gain more certainty by eventually asking about higher-level properties of the text, like "is it telling you to do something?".

Writing Python code to count the number of capital letters might seem straighforward, but it seems less clear how we can write Python code to determine whether the text is "telling you to do something".

We may not be able to write simple rules to determine if a piece of text is "telling you to do something"...but it sure sounds like the kind of task we would train a classifier for!

Layering Classifiers

Suppose we wanted a spam detection classifier that takes as input some high level features, such as:

• Is the text telling you to do something?
• Does the text contain "hype" language?
• Does the text sound scary?
• Does the text sound professional?

In other words, we want a classifier that looks kind of like this:

The key insight is that each of those features seems like the kind of thing we could train a classifier for! You can easily imagine, say, training a logistic regression model to classify text as professional-sounding or not. And so our idea is that instead of assuming the input features above are coming from the outside world, we can instead implement them as outputs from a previous layer of classifiers, like so:

One small implementation detail: notice we didn't label the "fancy features" in the middle as probabilities. This is because in practice, we actually don't want those middle outputs to be a probability distribution; more than one feature might be "active" at once (e.g., a text can be both scary and hype). So we wouldn't use the softmax activation function for those "intermediate classifiers"; we would use something simpler, like perceptron-style thresholding.

And yes, those "intermediate classifiers" could themselves depend on "fancy features" from a previous layer of classifiers; we can stack as many layers as we want. In general, this kind of multi-layer classifier has a name: it's known as a neural network.

To Be Continued...

We have more to say about neural networks, but this lesson has already run a bit long. So, in a somewhat rare move for CS 159, we'll cut things off here and then next week's online lesson will pick up where we left off.

(When logged in, completion status appears here.)

Previous Page Next Page