How to Train Your Neural Network

Neural Networks as Chains of Functions

When you hear mainstream media talk about neural networks, you often hear about the analogy to the human brain (which, as we noted last week, is kind of a sketchy comparison to begin with). Slightly more technical presentations may use fancy diagrams like the one we drew on the previous page, but the focus often still remains on the idea of a neural network as, well, a network of individual neurons.

But there is a much more useful way to think about neural networks, and that is as chains of functions.

In a neural network, each layer's output becomes the input to the next layer. Consider the simple 2-layer neural network we've been using as an example. For your easy reference, here it is again, with the two layers explicitly labeled:

Let's try to work out a mathematical formula for the output of this neural network. As usual, we'll designate this output as \( \hat{\mathbf{y}} \).

Notice that the output is produced by Layer 2. If we look at layer 2 in isolation, it just looks like a typical linear classifier, taking the "fancy features" (like "has instruction" and "has hype") as input. Let \( \mathbf W_{\text{layer2}} \) denote the weight matrix of the Layer 2 classifier, let \( \mathbf b_{\text{layer2}} \) be its bias vector, and let \( F_{\text{layer2}} \) be its activation function (which will probably be something like softmax, since Layer 2 is meant to output probabilities). Also, let \( \mathbf x_{\text{fancy}} \) denote the "fancy features" that Layer 2 takes as input.

Then, applying the standard classifier formula, we see that the output of Layer 2 can be written as:

(Equation 1) $$ \hat{\mathbf{y}} = F_{\text{layer2}}(\mathbf W_{\text{layer2}}\mathbf x_{\text{fancy}} + \mathbf b_{\text{layer2}}) $$

But now, of course, we come to the key question: what is \( \mathbf x_{\text{fancy}} \)? Again, conceptually this is meant to represent the "fancy features", like "has instruction" and "has hype", that the Layer 2 classifier takes as input. And remember that the whole premise of neural networks is that, as pictured in the diagram above, these "fancy features" can themselves be produced by a separate classifier—and that's exactly what Layer 1 is! In other words: \( \mathbf x_{\text{fancy}} \) is the output of Layer 1!

And since Layer 1 also looks like a classifier, with its own weights, biases, and activation function, we can similarly express \( \mathbf x_{\text{fancy}} \) using the standard classifier formula:

(Equation 2) $$ \mathbf x_{\text{fancy}} = F_{\text{layer1}}(\mathbf W_{\text{layer1}}\mathbf{x} + \mathbf b_{\text{layer1}}) $$

And now we just need to put all the pieces together! In (Equation 1), we can now substitute \( \mathbf x_{\text{fancy}} \) with the full definition of \( \mathbf x_{\text{fancy}} \) given in (Equation 2). Combining the equations like this gives us our final, complete equation for the output of the neural network:

$$ \hat{\mathbf{y}} = F_{\text{layer2}}(\mathbf W_{\text{layer2}}(F_{\text{layer1}}(\mathbf W_{\text{layer1}}\mathbf{x} + \mathbf b_{\text{layer1}})) + \mathbf b_{\text{layer2}}) $$

The key takeaway is this: for all the "mystery" surrounding neural networks as being "brain-like", the truth is they are more accurately understood as just chains of matrix operations. Bigger neural networks might have more chaining, but at the end of the day they can always be expressed as a series of matrix multiplications, additions, and other transformations. And it's important to build this insight because, once a neural network gets big enough, it's simply no longer practical to reason about it in terms of individual neurons...whereas reasoning about it in terms of math tends to scale better due to being more abstract.

Training a Neural Network

In order to build intuition, we labeled the intermediate outputs above with human-readable names. In reality, we don't actually have labels for anything other than the final output of the neural network (otherwise the amount of labeled data needed rise exponentially and become impractical). Inner layers of the neural network are best understood as "outputting some kind of intermediate features"; we don't actually know what those features are.

In light of this, how do we go about training a neural network? Surprisingly, it turns out the answer is...pretty much the same way we did before.

As noted above, a neural network is ultimately just a chain of matrix operations. And that means, no matter how scary it looks, you can still apply the chain rule to it!

This means that, just like linear classifiers, neural networks can be trained using gradient descent. Specifically, the backpropagation algorithm contains a clever trick to efficiently apply the chain rule one layer at a time, speeding up training.

In CS 159, we don't expect you to know anything about backpropagation other than the fact that it's just an application of the chain rule. Likewise, we don't need you to know how to actually implement neural network training; if you want to train a neural network for your final project, you can use libraries like PyTorch that have already implemented all the math for you. We do expect you to understand that, when you run the training logic in PyTorch, it's just doing gradient descent under the hood.

(When logged in, completion status appears here.)

Previous Page Next Page