Notebook
Tokens in, logits out
Build a byte-level BPE tokenizer, trace the path from token IDs to next-token loss, and see what weight tying changes.
The blog
Notes on
learning machines.
Foundation models, post-training, and the systems behind them. From first principles to working code.
I’m putting together three connected essays. Each will pair the core ideas with derivations, code, and controlled experiments.
01
In preparation
Inside a Modern LLM
From tokens to KV cache
Follow the tensors through tokenization, attention, a modern decoder, training, and cached generation.
- Tokens & embeddings
- Attention & decoder blocks
- Training & inference
This essay is in preparation.
02
In preparation
Building Foundation Models
Data, scale, and the move beyond text
A practical map of data pipelines, sparse models, efficient adaptation, multimodality, and evaluation.
- Data & scaling
- MoE & adaptation
- Multimodal systems
This essay is in preparation.
03
In preparation
Post-Training from First Principles
SFT, RLHF, DPO, and GRPO
Derive the objectives, implement the learning loops, and compare the methods under a shared evaluation setup.
- Supervised fine-tuning
- Preference learning
- Reinforcement learning
This essay is in preparation.
From the workbench
Start with something small.
Notebooks and visual explanations, alongside the longer essays.
Visual explanation
A closer look at activation functions
Sigmoid, GELU, SiLU, and softmax, side by side. An illustrated reference for the nonlinearities inside a neural network.