Implementation

Helpers

Let's start with helper functions that will be reused in multiple places.

Mathematical functions

First we need to introduce three mathematical functions: sigmoid, softmax, and cross_entropy. The implementation follows the formulas we defined earlier.

sigmoid takes an array of numbers and squashes each one into a value between 0 and 1.
softmax takes an array of scores and returns a probability distribution for each row.
cross_entropy takes the predicted probabilities and the correct indices, and returns an average loss.

Code

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def softmax(z):
    e = np.exp(z)
    return e / e.sum(axis=-1, keepdims=True)

def cross_entropy(probs, Y):
    correct = [probs[t, y] for t, y in enumerate(Y)]
    return -np.log(correct).mean()

# print("sigmoid:      ", sigmoid(np.array([-2.0])))
# 
# probs = softmax(np.array([[1.0, 0.1, 2.0]]))
# print("softmax:      ", probs)
# print("cross_entropy:", cross_entropy(probs, np.array([2])))

When the model scales up, some numbers can get too large or too small, so the formulas might need to be adjusted to handle these cases. This is not an issue for our small example, so we keep the raw formulas for simplicity.

Vocabulary

The vocabulary creates a list of the most common tokens in the given text and stores them with assigned ids.

tokenize splits the text into tokens (words and punctuation marks) using a regular expression.
encode turns the text into a sequence of token ids using the stoi (string-to-integer) lookup.
decode turns the sequence of token ids back into text using the itos (integer-to-string) lookup.

Code

import re
from collections import Counter

class Vocabulary:
    UNK = "<unk>"
    PAD = "<pad>"

    def __init__(self, text, size):
        special_tokens = [self.UNK, self.PAD]
        words = self._tokenize(text)

        counts = Counter(words)
        common_words = [
            word
            for word, _ in counts.most_common(size - len(special_tokens))
        ]

        self.itos = special_tokens + common_words
        self.stoi = {
            word: i
            for i, word in enumerate(self.itos)
        }

    def encode(self, text):
        unk = self.stoi[self.UNK]
        return [
            self.stoi.get(t, unk)
            for t in self._tokenize(text)
        ]

    def decode(self, ids):
        return " ".join(self.itos[i] for i in ids)

    def _tokenize(self, text):
        return re.findall(
            r"[a-z]+(?:'[a-z]+)?|[.,!?;:]", text.lower()
        )

# from implementation_overview import corpus, vocab_size

# vocab = Vocabulary(corpus, vocab_size)
# ids = vocab.encode("The little girl went to the garden.")

# print("ids:    ", ids)
# print("decoded:", vocab.decode(ids))

Training examples

For training we need to split the whole corpus into input/target pairs of size block that we will pass to our model.

Each row of X is a sequence of tokens in one example.
Each row of Y is a sequence of next tokens for each position in X row.

Code

def training_examples(ids, block):
    n = (len(ids) - 1) // block
    X = np.stack([
        ids[i * block:i * block + block]
        for i in range(n)
    ])
    Y = np.stack([
        ids[i * block + 1:i * block + block + 1]
        for i in range(n)
    ])
    return X, Y

# ids = [3, 4, 5, 6, 7, 8, 9, 10, 11]
# X, Y = training_examples(ids, 4)
# print("X:", X)
# print("Y:", Y)