Logistic regression

Our current model is called logistic regression and it is actually a very popular model for binary classification tasks. However, we need to make a small adjustment to it.

Let's write our model's formula again, but with some additional notation:

$$y = \sigma(z)$$

• $\sigma(z)$ is the notation for the sigmoid function, $\sigma(z)= \frac{1}{1 + e^{-z}}$
• $z$ is a new variable which is called logit. It should be calculated by another function. In our current model that function is $z=x$, so $\frac{1}{1 + e^{-z}}=\frac{1}{1 + e^{-x}}$

What we are currently missing for the complete logistic regression model is the correct function to calculate the logit. The complete formula for the logit calculation is:

$$z = w_0 + w_1 \cdot x_1 + w_2 \cdot x_2 + ... + w_n \cdot x_n$$

And for our case with only one input the logistic regression model will be:

$$y = \sigma(w_0 + w_1 \cdot x_1)$$

Parameters

We already know that $x_1, x_2, ..., x_n$ are the inputs (also known as features).

What are the $w_0, w_1, ..., w_n$? These are parameters (also known as weights). They give the model its flexibility. Depending on the values of parameters the model will perform differently.

If we draw the graph of the logistic regression function and mark our example messages on that graph, we can see how the model changes with different parameters.

With $w_0 = -20$ and $w_1 = 8$ we got the loss value close to 0 and 100% accuracy.

The model becomes:

$$y = \sigma(-20 + 8 \cdot x_1)$$

Scaling the model

We improved our model decreasing the loss value from 0.84 to 0 and increasing the accuracy from 40% to 100%.

However, it is very obvious that in reality the model is still bad. If we try to test it with random messages, we will probably not be satisfied in most cases.

The evaluation metrics tell us the model is good even though we know that it is bad. This is a bad sign meaning that we need to fix several things.

Splitting the datasets

Currently we adjust the parameters using the same dataset we use to evaluate the model. But to make a fair evaluation we should use different data for it which is called a training set.

Then we will have two different loss values:

• The error measured on a training set is called a training error and it measures how well we trained the model on that data.
• The error measured on a test set is called a generalisation error and it measures how the model performs on new data it has not seen.

Let's try to evaluate our current model on a completely new dataset:

Now the loss value for the generalisation error is 0.8, although the training error was close to 0.

Increasing the datasets

The issue when the generalisation error is high compared to the training error is called overfitting.

It happens when the model learns the training data too closely instead of learning the general patterns. It can be caused by several things, in our case it is because the training set is too small.

The whole idea of machine learning is to train a model on large datasets so it can learn patterns and perform well on new, unseen data. With a small training set our model can easily fit most of the messages but fails on new ones.

After increasing the datasets (900 messages in the training set and 100 messages in the test set) the best we can do is 0.64 as a training error and 0.61 as a generalisation error.

Adding more features

The errors are almost equal now, but the problem is they are both quite high. Well, at least now we have honest values for our current model's performance.

The issue when both of the errors are high is called underfitting. It is basically impossible for our current model to be good for all the examples, regardless of how we adjust the parameters.

That is because our model is too simple.

One way we can improve it is to use more features counting other words and symbols beside !:

The model formula will be:

$$y = \sigma(w_0 + w_1 x_1 + w_2 x_2 + ... + w_{30} x_{30})$$

With adjusted parameters for 30 features the model achieves 0.11 as a training error and 0.14 as a generalisation error.

Much better than before. But how exactly do we adjust these parameters for 30 features and 1000 messages?