Unblock all features
Sign in to track progress, get AI help and save conversations.
Let's start with helper functions that will be reused in multiple places.
First we need to introduce three mathematical functions: sigmoid, softmax, and cross_entropy.
The implementation follows the formulas we defined earlier.
sigmoid takes an array of numbers and squashes each one into a value between 0 and 1.softmax takes an array of scores and returns a probability distribution for each row.cross_entropy takes the predicted probabilities and the correct indices, and returns an average loss.When the model scales up, some numbers can get too large or too small, so the formulas might need to be adjusted to handle these cases. This is not an issue for our small example, so we keep the raw formulas for simplicity.
The vocabulary creates a list of the most common tokens in the given text and stores them with assigned ids.
tokenize splits the text into tokens (words and punctuation marks) using a regular expression.encode turns the text into a sequence of token ids using the stoi (string-to-integer) lookup.decode turns the sequence of token ids back into text using the itos (integer-to-string) lookup.For training we need to split the whole corpus into input/target pairs of size block that we will pass to our model.
X is a sequence of tokens in one example.Y is a sequence of next tokens for each position in X row.