The Neural Network Zoo

The neural network we showed you just now has two key properties: each layer resembles a traditional linear classifier, and each layer's output is strictly fed only to the next layer. This is actually a very specific type of neural network, called a feed-forward neural network. It's called that because each layer, well, feeds forward into the next one.

But feed-forward networks are only one of many types of neural networks. While we won't go over the math for other types of neural networks, we do want to quickly summarize the high-level properties of some of the most popular neural network types, so that you know what options are available to you.

Convolutional Neural Networks

These are probably the most similar to basic feed-forward networks. The main difference is that, while a feed-forward layer contains a weight matrix that applies to the entire input vector, a convolutional layer contains a smaller weight matrix which is applied to chunks of the input vector one chunk at a time, in a sliding window fashion (imagine taking a piece of glass and sliding it across a ruler). This architecture was loosely inspired by how vision works in animals (including humans) and therefore is most successful on images. Convolutional networks are generally considered less effective on text and are therefore less popular in NLP, though you might still see them from time to time.

Recurrent Neural Networks

We already briefly mentioned recurrent neural networks (RNNs) in Week 1, but now we have the conceptual foundation to discuss them in slightly more detail. Basically, the main difference between an RNN and a feed-forward network is that we drop the "feed-forward" part, and allow a layer's output to instead feed backward in a feedback loop to a previous layer.

For many years, the challenge in making RNNs practical was solving the problem of positive feedback explosions, where the feedback loop causes values to just keep accumulating, causing the output to get larger and larger until we hit a integer or floating point overflow (i.e., it "explodes"). The eventual solution was a concept known as gating. At a very high level, you can think of a "gating unit" as a small classifier that, for each output, decides whether or not that output should actually be fed back to a previous layer. As this description implies, the simplest possible gating mechanism would just be a binary yes/no, but more advanced approaches could instead apply a "discount factor" that multiplies the output by some small but non-zero value, achieving a similar effect.

Two gating-based approaches to RNNs are long short-term memory (LSTM) and gated recurrent units (GRU).

RNNs form the basis of what was once a very popular approach to neural network language modeling: the sequence-to-sequence model. In general, a sequence-to-sequence model maps an input sequence \( x \) to an output sequence \( y \). One classic example is machine translation, where the input sequence is words in one language (say, English) and the output sequence is words in another language (say, Chinese) such that the input and output have roughly the same meaning. RNNs are useful for this task because, conceptually, you have to remember what you previously generated so that you don't lose your position in the output sequence. Gating units like LSTMs and GRUs can be thought of as a primitive form of memory!

Transformers

Transformers are the technology behind modern Large Language Models. We will discuss transformers in class after Fall Break, so we won't talk too much about them here. Here's the general intuition: notice that a feed-forward network doesn't really permit much interaction between features. The way the dot product works, each feature just gets multiplied with its own corresponding weight; the presence of some other feature somewhere else cannot affect this. This turns out to be a major limitation in NLP, because the meaning of one word often actually depends on other words around it. One obvious example is pronouns like "it"; what "it" refers to depends on what nouns showed up before it in the document.

The transformer fixes this limitation by introducing a new mathematical operation known as attention. As we will discuss in class, attention provides a way to score, for any given feature, how important each of the other features is.

Choosing the Right Tool for the Job

Once again, we want to reiterate that you're not expected to know too much about the details of neural networks in CS 159. And you're certainly not expected to know the math behind things like convolutional or recurrent neural networks, which we never even explained. We mostly just want you to know what options are out there, in case you might want to use them for your final project.

In practice, projects that do end up using neural networks tend to fall into two broad categories:

1. If you want to train a classifier from scratch on a small, curated dataset, you will probably end up using feed-forward networks (or something even simpler like logistic regression). These are the easiest networks to train (you might not even need a GPU!), and for text classification you probably don't need something much more powerful.
2. If you want to do something with LLMs, you will (indirectly) be using transformers. You probably can't train an LLM from scratch, but you can download an off-the-shelf model and run it on the server's GPU. You can customize off-the-shelf LLMs either by fine-tuning them on your own training data, or just writing custom prompts.

(When logged in, completion status appears here.)

Previous Page Next Page