Inside a Modern LLM: From Tokens to KV Cache

A bottom-up guide to decoder-only language models, with derivations, PyTorch implementations, ablations, and systems measurements.

On this page

• 1 What the reader will build
• 2 Chapter 1 · Tokens in, logits out
• 3 Chapter 2 · Self-attention from first principles
• 4 Chapter 3 · The modern decoder block
• 5 Chapter 4 · Training a Tiny GPT
• 6 Chapter 5 · Decoding and KV cache
• 7 Chapter 6 · Anatomy of real models
• 8 Reproducibility checklist

Thesis. A language model stops being mysterious once every boundary—tokens, tensors, weights, logits, loss, cache, and samples—can be traced and measured.

Note

This is a living essay under active development. Each chapter is backed by a reproducible notebook and will be expanded during the 18-day sprint.

1 What the reader will build

• A tokenizer and input/output embedding path
• Multi-head causal self-attention from tensor operations
• A modern decoder block with RMSNorm, RoPE, GQA, and SwiGLU
• A trainable Tiny GPT
• Cached autoregressive decoding and a latency/memory benchmark
• A model-anatomy script for inspecting real open-source checkpoints

2 Chapter 1 · Tokens in, logits out

2.1 Historical problem

2.2 Tensor contract

2.3 Mathematics

2.4 Implementation

2.5 Ablation and visualization

2.6 Interview takeaways

3 Chapter 2 · Self-attention from first principles

3.1 From additive attention to Q/K/V

3.2 Mathematics and dimensions

3.3 Implementation and tests

3.4 Ablation and visualization

4 Chapter 3 · The modern decoder block

4.1 Architecture evolution

4.2 Components and parameter ledger

4.3 Implementation

4.4 Ablation and visualization

5 Chapter 4 · Training a Tiny GPT

5.1 Objective, data, and optimization

5.2 End-to-end implementation

5.3 Ablation and visualization

5.4 Failure analysis

6 Chapter 5 · Decoding and KV cache

6.1 Prefill versus decode

6.2 Cache mathematics

6.3 Implementation

6.4 Systems benchmark

7 Chapter 6 · Anatomy of real models

7.1 What is public and what is not

7.2 Config-to-weights inspection

7.3 Model-family comparison

7.4 Final mental model

8 Reproducibility checklist

• Colab links run from a clean runtime
• Package/model/data versions are pinned
• Random seeds and hardware are reported
• Plots include axes, units, and sample counts
• Cached and uncached logits have a numerical equivalence test
• Closed-model claims are sourced and labeled
• Failed experiments are documented