Blog

Notes & deep dives

Long-form, intuition-first writeups: the idea in plain language first, then the math, a worked example, and runnable code. Mostly reinforcement learning and LLM systems for now.

• May 5, 2026

  Reinforcement Learning from First Principles (and the Math You Actually Need)

  What RL really is, why 'how good' means 'how good on average', and the five pieces of math (expectation, the Markov property, variance, running averages, and discounting) that the rest of the field is built on. Intuition first, one clean derivation each, a worked example, then code with Gymnasium.

  • #reinforcement-learning
  • #fundamentals
  • #probability
  • #gymnasium

• May 12, 2026

  MDPs and the Bellman Equation: The Recursion Behind Every RL Algorithm

  How to formalize an environment as a Markov Decision Process, why the value of a state splits into V and Q, and the Bellman equation that ties them together recursively. Derive it once, verify it by hand, then solve it exactly on FrozenLake in Gymnasium.

  • #reinforcement-learning
  • #mdp
  • #bellman-equation
  • #gymnasium

• May 19, 2026

  DP, Monte Carlo, and TD: Three Ways to Solve the Bellman Equation

  Three algorithms (Dynamic Programming, Monte Carlo, and TD(0)) all solve the same Bellman equation. You'll implement each on a custom Mars Rover gridworld and watch them converge to the same values from different starting assumptions.

  • #reinforcement-learning
  • #dynamic-programming
  • #monte-carlo
  • #temporal-difference
  • #gymnasium

• May 26, 2026

  SARSA, Q-learning, and DQN: From a Table to a Network That Plays Atari

  Control = prediction + a max. We turn the TD update into Q-learning and SARSA, watch them disagree on a cliff, then swap the table for a neural network and train a DQN that learns Pong from raw pixels.

  • #reinforcement-learning
  • #q-learning
  • #sarsa
  • #dqn
  • #deep-rl
  • #gymnasium

• Jun 2, 2026

  Policy Gradients: Learning the Policy Directly, from a Bandit to Actor-Critic

  Stop learning values and reading a policy off them: parameterize the policy and climb expected reward directly. We derive the score-function trick on a one-state Archer bandit, watch one softmax update touch all nine logits, kill the variance with a baseline, add states and credit assignment, and end on an Actor-Critic that solves the Archer MDP.

  • #reinforcement-learning
  • #policy-gradients
  • #reinforce
  • #actor-critic
  • #advantage
  • #gymnasium

• Jun 9, 2026

  TRPO and PPO: The Largest Safe Step a Policy Can Take

  The gradient gives a direction, but who chooses the step size? This post explores the step-size crisis in RL, builds importance sampling from scratch, derives the surrogate objective, fences it with TRPO's KL constraint, then clips it with PPO, all grounded in a continuous-control example where PPO learns to swing a pendulum upright.

  • #reinforcement-learning
  • #ppo
  • #trpo
  • #importance-sampling
  • #trust-region
  • #gymnasium

• Jun 16, 2026

  RLHF: Teaching a Language Model What "Good" Means

  A language model only ever predicts the next token. This post earns the sentence "PPO aligned ChatGPT": how generating text becomes an MDP, how human comparisons become a Bradley-Terry reward model, how a value head turns one end-of-answer score into a per-token signal, and how the KL leash keeps PPO from gaming a flawed reward, grounded in a hands-on build that aligns a tiny GPT-2 assistant.

  • #reinforcement-learning
  • #rlhf
  • #reward-model
  • #bradley-terry
  • #ppo
  • #kl-penalty

• Jun 23, 2026

  GRPO: Teaching a Model to Reason by Comparing It to Itself

  RLHF aligned a model with two heavy passengers: a learned reward model and a value-head critic. This post throws both away. You will see why the critic was only ever a baseline and how a group of sampled answers replaces it for free, why a verifiable check beats a hackable reward model for math and code, how GRPO wraps that group-relative advantage in PPO's clip, and how this exact recipe produced DeepSeek-R1, all grounded in a hands-on run that trains Llama-3.2-3B to reason on grade-school math.

  • #reinforcement-learning
  • #grpo
  • #verifiable-rewards
  • #rlvr
  • #deepseek-r1
  • #reasoning
  • #ppo

• Jun 30, 2026

  DPO and Agentic RL: Align Without a Reward Model, Then Step Into the World

  Two moves close the series. First we simplify: Direct Preference Optimization reaches the same aligned model as RLHF while firing both the reward model and the RL loop, and SimPO and KTO trim it further. Then we expand: point the same group-relative gradient at a multi-step world and the model stops answering and starts acting, calling tools, reading observations, and earning a reward only at the end. You will see the DPO loss derived from the RLHF objective, every number of a worked preference, and a runnable multi-turn agent that learns to use a tool from reward alone.

  • #reinforcement-learning
  • #dpo
  • #simpo
  • #kto
  • #preference-optimization
  • #agentic-rl
  • #rl-environments
  • #llm

• Jul 15, 2026

  Socratic Alignment: Teaching a Small Model to Withhold the Answer

  Every alignment method you know pushes a model to say more. This post runs the machinery backwards: seven recipes (SFT, DPO, KTO, ORPO, SimPO, GRPO, PPO) all try to teach a 0.5B model to hold the answer back and guide with a question instead. You will build the classroom that makes teaching measurable, a three-layer scorecard from a ten-line leakage rule to a real learning-gain, and then read the results honestly: anchored recipes teach, pure contrast evades, and the online RL heavyweights barely move.

  • #reinforcement-learning
  • #alignment
  • #preference-optimization
  • #reward-hacking
  • #llm-judge
  • #orpo
  • #rlvr
  • #llm

• Jul 16, 2026

  Dreaming to Dodge: Training a Doom Agent Entirely Inside Its Own Dream

  Build the three components of an IRIS world model from scratch (a VQ-VAE tokenizer, a GPT-style Transformer, a tiny evolved controller) and train an agent to survive VizDoom take_cover without ever touching the real game. Along the way, watch every famous model-based RL failure happen for real: the tokenizer erases the fireball, the model never dreams death, and the policy learns to cheat the dream. The final agent beats a model-free DQN at the same real-frame budget and matches a hand-coded oracle.

  • #reinforcement-learning
  • #world-models
  • #model-based-rl
  • #vq-vae
  • #transformers
  • #iris
  • #vizdoom

• Jul 20, 2026

  Grasping in the Dark: A Robot Arm Learns to Pick Up a Cube from a Reward It Almost Never Sees

  Reproduce the learning core of HIL-SERL (SAC + RLPD) on a simulated Franka Panda: a sparse-reward grasp learned from 0% to ~100% success in about 40 minutes on a single cloud GPU, for roughly $0.55 of compute. Derive the soft Bellman target, see exactly why a 50/50 blend of fresh experience and 30 old demonstrations turns an invisible reward into a learnable one, and then follow the same code onto a real $200 SO-101 arm, where four undocumented walls and one normalization bug are waiting.

  • #reinforcement-learning
  • #robotics
  • #sac
  • #rlpd
  • #hil-serl
  • #lerobot

No posts with that tag yet.