Overview

Now we will implement in code the exact same architecture that was described in the previous chapters.

The whole implementation is Python code with the numpy library only. We will not use any neural network frameworks as our main goal is to understand how it works under the hood.

All the code that appears in this section is editable and runnable. But we will need to use a smaller corpus and smaller hyperparameters to be able to run it in the browser.

Here is the full list of parameters that we declare upfront. For our code examples running in the browser we will use the default values. If you want to experiment with scaling the model you can copy the full code and run it locally, tweaking these parameters.

We do not introduce parameters for number of attention heads and transformer blocks as handling just one head and one transformer block simplifies the code.

It is worth mentioning though, even with a fully parameterised version of this model there are still limitations on how much we can scale it. If we want to reach frontier-model behaviour there is a bit more to it than just tweaking these parameters.

Real LLMs run on GPUs, the training is distributed on multiple machines, and there are optimisation algorithms that we skip entirely in the code. All of these would be essential if we scale it to trillions of parameters as it becomes huge.

Another thing that we skipped is post-training (fine-tuning and RLHF). It is needed to give an LLM assistant-like behaviour by training on additional data in question/answer format.

We will skip all of these in our code examples, as they do not change the core architecture of the model that we want to understand.