Key Points
- The major drawback of Naive Bayes is that it is not robust to cases where multiple features might be strongly correlated.
- However, there are other types of linear classifiers that can do better in such cases.
- One type of linear classifier is the perceptron, which takes a weighted sum of input features and applies a threshold to reach a binary (yes-or-no) output.
- The weighted sum step is common to all linear classifiers, and can be expressed as a dot product between the input features and a weight vector.
- For classification across multiple classes, you can have one weight vector for each class; this can be cleanly expressed as a matrix multiplication.
- The thresholding done in the perceptron is just one example of an activation function.
- Another example of an activation function is the softmax function, which turns a vector of per-class predictions into a valid probability distribution.
- Linear classifiers can be trained using gradient descent, an application of the chain rule of calculus.
- A linear classifier that uses softmax activation and logistic loss is called logistic regression.
- A neural network effectively consists of multiple layers that resemble individual linear classifiers, where each layer's output is fed to the next layer as input.
Takeway Skills
On a subsequent lab assignment or exam, you are expected to be able to:
- Define the bolded key terms above.
- Given a set of scenarios, identified based on properties of the features, identify scenarios that Naive Bayes would be bad at.
- Execute a simple perceptron example: given the weights, bias terms, and an input feature vector, determine whether the perceptron will output 1 or 0 for that input.
- Explain the relationship between logistic regression and the perceptron.
- Explain the relationship between neural networks and linear classifiers.
(When logged in, completion status appears here.)