Neural Networks

Logistic regression model has an important limitation. It is a linear model. Each feature is multiplied by some constant parameter and the results are just summed up into the logit.

$$z = w_0 + w_1 \cdot x_1 + w_2 \cdot x_2 + ... + w_n \cdot x_n$$

That means a couple of things:

• The effect of a feature on the logit is always strictly proportional to its value. If one character adds 2 to the logit, ten characters add 20. However, more characters don't always mean the message is more likely spam.
• The effect of a feature on the logit is isolated. If we have 2 occurrences of free, it always adds the same value to the logit. However, free in isolation might not be that spammy compared to having both free and money in the same message.

Decision boundary

To make it easier to illustrate the non-linearity problem, let's look at a model with one feature (number of characters) and a small number of training examples.

In this training set, the messages are spam when they are too short or too long, and not spam when they have a medium length.

We can't fit this training set with our model, because the decision boundary is always a single point, regardless of how we change the parameters.

So the maximum accuracy we can reach on this dataset is only 67%. This is another cause of underfitting.

We could make the model more flexible by manually changing the logit formula. E.g. square some features ($x_1^2$) or multiply together ($x_1 \cdot x_2$). This technique is called feature engineering.

However, it does not scale well, with more features the number of combinations explodes, especially for large models like LLMs. Instead, we will use neural networks to address these issues.

Neural network model

The core idea is to stack many simple functions on top of each other.

For example, we can take our current model and make three different logistic regression functions instead of one:

$$\begin{aligned} a_1^{[1]} &= \sigma\big(w_{1,0}^{[1]} + x_1w_{1,1}^{[1]} + x_2w_{1,2}^{[1]} + \dots + x_{30}w_{1,30}^{[1]}\big) \\ a_2^{[1]} &= \sigma\big(w_{2,0}^{[1]} + x_1w_{2,1}^{[1]} + x_2w_{2,2}^{[1]} + \dots + x_{30}w_{2,30}^{[1]}\big) \\ a_3^{[1]} &= \sigma\big(w_{3,0}^{[1]} + x_1w_{3,1}^{[1]} + x_2w_{3,2}^{[1]} + \dots + x_{30}w_{3,30}^{[1]}\big) \end{aligned}$$

And then combine the results into another final function:

$$y = \sigma\big(w_{1,0}^{[2]} + a_1^{[1]}w_{1,1}^{[2]} + a_2^{[1]}w_{1,2}^{[2]} + a_3^{[1]}w_{1,3}^{[2]}\big)$$

This is a neural network with four neurons and two layers.

The number of inputs and outputs is the same, but we have more parameters now. So we use $w_{jk}^{[l]}$ notation to describe different parameters on different neurons:

• $l$ is the layer of the neuron
• $j$ is the neuron's index in the current layer
• $k$ is the feature's index or the index of the input neuron from the previous layer

With the neural network the function graph is more flexible, we have two decision boundaries now and can reach 100% accuracy on the same dataset.

Training the neural network

The training algorithm is the same, but we have more derivatives to calculate now. The loss function is bigger as well, since it consists of multiple functions stacked across neurons.

We should apply the same chain rule to calculate the derivatives with respect to each parameter. For example for our small neural network with two layers:

• The output neuron derivatives over each parameter will be:

$$\frac{dL}{dw_{1,k}^{[2]}} = \frac{dL}{d\hat{y}} \cdot \frac{d\hat{y}}{dz_1^{[2]}} \cdot \frac{dz_1^{[2]}}{dw_{1,k}^{[2]}}$$

• The hidden layer neurons derivatives over each parameter will have a longer chain:

$$\frac{dL}{dw_{j,k}^{[1]}} = \frac{dL}{d\hat{y}} \cdot \frac{d\hat{y}}{dz_1^{[2]}} \cdot \frac{dz_1^{[2]}}{da_j^{[1]}} \cdot \frac{da_j^{[1]}}{dz_j^{[1]}} \cdot \frac{dz_j^{[1]}}{dw_{j,k}^{[1]}}$$

Testing the model

Our simple neural network is called a Multilayer Perceptron (MLP) where every neuron is connected to every neuron in the next layer.

Using the neural network the accuracy increased to 96%. Here is how the model behaves for any text message:

There are still a lot of potential improvements we could make. For example: using more data, using more features, presenting inputs differently, scaling the neural network. But these are just implementation details that do not change the core idea.

So we will leave the spam filtering model as it is and return to our original problem in the next section: building an LLM.