ML fundamentals

Evaluation

The first version of our model is bad, the predictions probably will not match the real values in many cases. But how bad is it? Can we measure it?

Test set

We can measure the model's performance if we have a test set data to compare with. Imagine we have a dataset where humans manually reviewed and labeled each message.

Message	Label
Can you review my report?	not spam
Thanks for your help!	not spam
Happy birthday!!	not spam
FREE MONEY! CLAIM IT NOW!!	spam
CLICK HERE!! LIMITED OFFER!!	spam

Accuracy

Now we can compare the model's predictions with the real labels from the test set for each message. The model is good when the results are the same and bad when they are different.

The metric that does it is called accuracy. The accuracy is a percentage of correct predictions over the whole dataset.

\text{accuracy}=\frac{\text{correct predictions}}{\text{total predictions}} \cdot 100

We consider the prediction as not spam if the predicted spam score is less than 0.5 (the threshold) and as spam otherwise.

For our current model $y = \frac{1}{1 + e^{-x}}$ , the accuracy for the above test set is 40%.

This is how our model is currently performing, and our goal is to improve it.

Loss function

The other way to evaluate the model is using a loss function. It measures how wrong the actual predicted values are instead of collapsing them to yes/no labels.

There are different formulas for the loss function. For our binary classification task we will use cross-entropy:

L = -\frac{1}{n} \sum_{i=1}^{n} \big[ y_i \log_e(\hat{y}_i) + (1 - y_i) \log_e(1 - \hat{y}_i) \big]

$y_i$ is the real spam label of the $i$ -th message (from the test set).
$\hat{y}_i$ is the predicted spam score of the $i$ -th message (our model's result).
$n$ is the total number of messages in the test set.

It is basically the average of losses for each test case. For each message the formula collapses and only one of two terms is left:

If the message is spam ( $y_i = 1$ ), the $(1-y_i) \cdot \log_e(1 - \hat{y}_i)$ part becomes zero. The loss becomes:

L_i=- \log_e(\hat{y}_i)

If the message is not spam ( $y_i = 0$ ), the $y_i \log_e(\hat{y}_i)$ part becomes zero. The loss becomes:

L_i =- \log_e(1-\hat{y}_i)

The minus sign is needed to make the result always positive as $\log_e$ of a positive number smaller than 1 is always negative.

The $\log_e$ ensures the errors are punished more when the model is confidently wrong. E.g.

predicting a big spam score ( $\hat{y}_i = 0.99$ ) when the message is not spam ( $y_i = 0$ ) will result in a big loss value as

L_i = - \log_e(0.01) = 4.6

predicting a small spam score ( $\hat{y}_i = 0.1$ ) when the message is not spam ( $y_i = 0$ ) will result in a small loss value as

L_i = - \log_e(0.99) = 0.01

For our current model $y = \frac{1}{1 + e^{-x}}$ , the loss value for the above test set is 0.84.

We will use both the loss function and accuracy to measure the model.

The accuracy is simple, and it is easy to intuitively understand how the model performs.

The loss function is more detailed, and we will need these details to improve our model.