Introduction

This is my attempt to answer a single question: How do Large Language Models (LLMs) work under the hood?

I have tried to make the answer concise and complete. Well, at least as concise and complete as possible.

We can't cover every aspect of gigantic models like ChatGPT or Claude, but we will deeply cover the main mechanism that allows them to generate responses.

In fact, we will build our own small LLM from scratch, which should give us an end-to-end understanding of the main concept down to every single computation.

We will do it in three parts. In the first two parts we will build the mathematical formulation of our model, then implement it in code in the third.

Even though we will need to go through some theory on ML fundamentals, this is not a full course on AI, ML or Deep Learning. Our focus is narrow: understand the concept of an LLM and build it.