Post-Training from First Principles: SFT, RLHF, DPO and GRPO

A derivation-first, implementation-driven map of modern post-training.

On this page

• 1 What the reader will build
• 2 Chapter 1 · From a base model to an assistant
• 3 Chapter 2 · Preference data and reward models
• 4 Chapter 3 · RLHF and PPO from first principles
• 5 Chapter 4 · DPO and direct preference optimization
• 6 Chapter 5 · Reasoning RL and GRPO
• 7 Chapter 6 · Evaluation, safety, and research judgment
• 8 Reproducibility checklist

Note

This is a living essay under active development. Its central artifact is a controlled comparison using the same base model, data split, and evaluation harness.

Thesis. Post-training is a sequence of distribution-shaping problems: supervised imitation, preference estimation, constrained policy optimization, and evaluation under imperfect rewards.

1 What the reader will build

• A QLoRA supervised fine-tuning baseline
• A pairwise preference dataset and reward model
• A miniature PPO/RLHF loop with explicit KL control
• DPO implemented directly from its loss equation
• A small GRPO experiment on verifiable tasks
• A unified evaluation report and model card

2 Chapter 1 · From a base model to an assistant

2.1 History and theory

2.2 Mathematics

2.3 Colab implementation

2.4 Ablation and visualization

2.5 Failure analysis

3 Chapter 2 · Preference data and reward models

3.1 History and theory

3.2 Mathematics

3.3 Colab implementation

3.4 Ablation and visualization

4 Chapter 3 · RLHF and PPO from first principles

4.1 History and theory

4.2 Mathematics

4.3 Colab implementation

4.4 Ablation and visualization

5 Chapter 4 · DPO and direct preference optimization

5.1 History and theory

5.2 Derivation

5.3 Colab implementation

5.4 Ablation and visualization

6 Chapter 5 · Reasoning RL and GRPO

6.1 History and theory

6.2 Mathematics

6.3 Colab implementation

6.4 Ablation and visualization

7 Chapter 6 · Evaluation, safety, and research judgment

7.1 Evaluation system

7.2 Safety and red-teaming

7.3 Unified comparison

7.4 Model card and research proposals

8 Reproducibility checklist

• All checkpoints share the same documented base model
• Train and evaluation data are separated and decontaminated
• Preference formatting and chat templates are identical across methods
• Reward, KL, entropy, length, and task metrics are logged together
• Ablations change one primary variable at a time
• Failed or unstable runs are included
• Safety and capability regressions are both reported
• The model card distinguishes evidence from speculation