ML Model

I want us to build the model iteratively, starting with the simplest working version and gradually improving it.

However, LLMs rely on multiple core concepts at the same time, making it hard to immediately build it. That is why we will start with a simpler problem in this section: spam filtering.

Our model will take a message as input and decide whether it is spam or not. This type of task is called binary classification.

Once we build this foundation, we will extend the model in the next section, turning it into an LLM that generates text.

Let's start!

Input and output

First, we need to represent input and output as numbers.

For the output we can return a spam score between 0 and 1 which is a probability of a message being spam. The bigger the number is, the more likely the message is spam.

For the input we can count each word and each punctuation mark in the text, or at least some of them. This technique is called bag of words. For now let's start counting only the number of exclamation marks (!).

For example, in the text FREE MONEY!!! CALL ASAP! the input will be 4 as we have four !.

Sigmoid function

As we have the input and output numbers, the model is just a mathematical function:

$$f(x) = y$$

The function takes the input, does some computation, and produces the output.

For our case we have one input:

• $x$ for the number of !

and one output:

• $y$ for the spam score

What is left is a computation. Keeping in mind that the output of the computation should always be a number between 0 and 1 we can use this formula:

$$y = \frac{1}{1 + e^{-x}}$$

This function is called a sigmoid function. It is often used in binary classification tasks, because it turns any number into a probability. $y$ will always stay between 0 and 1, regardless of $x$.

This is the first version of our model.

Replacing $x$ with 4 (the number of ! in our example):

$$y = \frac{1}{1 + e^{-4}} \approx 0.98$$

The model predicts a spam score of 0.98 for the text "FREE MONEY!!! CALL ASAP!".

Inference

We can use the model now to make predictions for other examples. This is called inference.

Inference itself is a big topic because in production there can be different issues like performance, efficiency, etc. But we will not cover it in much detail, as for our small model it is just putting the inputs into the function and returning the output.

Here is how the first version of our model behaves for any text message. Note that it is not accurate for now, but at least we have something already to work with and improve: