https://prathameshsaraf.com/blogs/ I help enterprise teams ship production GenAI: voice agents, agentic workflows, RAG, and the infrastructure to make them stick. en https://prathameshsaraf.com/blogs/inference-04-flashattention-2/ https://prathameshsaraf.com/blogs/inference-04-flashattention-2/ Tue, 01 Sep 2026 00:00:00 GMT FlashAttention 1 solved the memory wall and still left three quarters of the tensor cores idle. The fix was not a better algorithm but an inventory of where the chip was waiting: bookkeeping on the wrong cores, too few thread blocks to fill the machine, and warps trading partial results they never should have held. Three rearrangements roughly double the fraction of peak, and then Hopper arrives and takes most of it back. llm-inference flash-attention kernels gpu occupancy warps scheduling https://prathameshsaraf.com/blogs/inference-03-kernels-and-flashattention/ https://prathameshsaraf.com/blogs/inference-03-kernels-and-flashattention/ Thu, 20 Aug 2026 00:00:00 GMT Two kernels can compute identical attention on the same GPU and finish 7.6 times apart. The arithmetic cannot be the difference, so the whole gap lives in data movement. This post builds FlashAttention from first principles: why the N-by-N score matrix is the problem, how tiling keeps it off the memory bus, and how online softmax makes tiling legal, and why the result is exact rather than an approximation. llm-inference flash-attention kernels triton attention softmax gpu https://prathameshsaraf.com/blogs/inference-02-inside-the-gpu/ https://prathameshsaraf.com/blogs/inference-02-inside-the-gpu/ Fri, 14 Aug 2026 00:00:00 GMT The last post found the bottleneck. This one opens the machine that creates it. You will see why a GPU spends its silicon on 270,000 slow threads instead of a few fast ones, why a trip to HBM costs 482 cycles while a register costs 1, how the chip hides that wait instead of avoiding it, and what two programming mistakes throw the whole advantage away. It ends on a controlled experiment Nvidia effectively ran for us: the same compute die with faster memory, and what that proves about inference. llm-inference gpu cuda tensor-cores memory-hierarchy quantization nvlink systems https://prathameshsaraf.com/blogs/inference-01-memory-wall/ https://prathameshsaraf.com/blogs/inference-01-memory-wall/ Mon, 10 Aug 2026 00:00:00 GMT A frontier model emits an output token about every 30 milliseconds, but the arithmetic inside that token takes a tenth of a millisecond. This post accounts for the other 99 percent. You will derive an H100's ridge point of 295 operations per byte, place prefill and decode on a roofline 2,000x apart, compute a KV cache down to the byte, and finish able to predict any model's decode speed on any chip from just two numbers. llm-inference memory-wall roofline kv-cache gpu transformers systems https://prathameshsaraf.com/blogs/09-dpo-and-agentic-rl/ https://prathameshsaraf.com/blogs/09-dpo-and-agentic-rl/ Tue, 30 Jun 2026 00:00:00 GMT Two moves close the series. First we simplify: Direct Preference Optimization reaches the same aligned model as RLHF while firing both the reward model and the RL loop, and SimPO and KTO trim it further. Then we expand: point the same group-relative gradient at a multi-step world and the model stops answering and starts acting, calling tools, reading observations, and earning a reward only at the end. You will see the DPO loss derived from the RLHF objective, every number of a worked preference, and a runnable multi-turn agent that learns to use a tool from reward alone. reinforcement-learning dpo simpo kto preference-optimization agentic-rl rl-environments llm https://prathameshsaraf.com/blogs/08-grpo/ https://prathameshsaraf.com/blogs/08-grpo/ Tue, 23 Jun 2026 00:00:00 GMT RLHF aligned a model with two heavy passengers: a learned reward model and a value-head critic. This post throws both away. You will see why the critic was only ever a baseline and how a group of sampled answers replaces it for free, why a verifiable check beats a hackable reward model for math and code, how GRPO wraps that group-relative advantage in PPO's clip, and how this exact recipe produced DeepSeek-R1, all grounded in a hands-on run that trains Llama-3.2-3B to reason on grade-school math. reinforcement-learning grpo verifiable-rewards rlvr deepseek-r1 reasoning ppo https://prathameshsaraf.com/blogs/07-rlhf/ https://prathameshsaraf.com/blogs/07-rlhf/ Tue, 16 Jun 2026 00:00:00 GMT A language model only ever predicts the next token. This post earns the sentence "PPO aligned ChatGPT": how generating text becomes an MDP, how human comparisons become a Bradley-Terry reward model, how a value head turns one end-of-answer score into a per-token signal, and how the KL leash keeps PPO from gaming a flawed reward, grounded in a hands-on build that aligns a tiny GPT-2 assistant. reinforcement-learning rlhf reward-model bradley-terry ppo kl-penalty https://prathameshsaraf.com/blogs/06-trpo-ppo/ https://prathameshsaraf.com/blogs/06-trpo-ppo/ Tue, 09 Jun 2026 00:00:00 GMT The gradient gives a direction, but who chooses the step size? This post explores the step-size crisis in RL, builds importance sampling from scratch, derives the surrogate objective, fences it with TRPO's KL constraint, then clips it with PPO, all grounded in a continuous-control example where PPO learns to swing a pendulum upright. reinforcement-learning ppo trpo importance-sampling trust-region gymnasium https://prathameshsaraf.com/blogs/05-policy-gradients/ https://prathameshsaraf.com/blogs/05-policy-gradients/ Tue, 02 Jun 2026 00:00:00 GMT Stop learning values and reading a policy off them: parameterize the policy and climb expected reward directly. We derive the score-function trick on a one-state Archer bandit, watch one softmax update touch all nine logits, kill the variance with a baseline, add states and credit assignment, and end on an Actor-Critic that solves the Archer MDP. reinforcement-learning policy-gradients reinforce actor-critic advantage gymnasium https://prathameshsaraf.com/blogs/04-sarsa-qlearning-dqn/ https://prathameshsaraf.com/blogs/04-sarsa-qlearning-dqn/ Tue, 26 May 2026 00:00:00 GMT Control = prediction + a max. We turn the TD update into Q-learning and SARSA, watch them disagree on a cliff, then swap the table for a neural network and train a DQN that learns Pong from raw pixels. reinforcement-learning q-learning sarsa dqn deep-rl gymnasium https://prathameshsaraf.com/blogs/03-dp-mc-td/ https://prathameshsaraf.com/blogs/03-dp-mc-td/ Tue, 19 May 2026 00:00:00 GMT Three algorithms (Dynamic Programming, Monte Carlo, and TD(0)) all solve the same Bellman equation. You'll implement each on a custom Mars Rover gridworld and watch them converge to the same values from different starting assumptions. reinforcement-learning dynamic-programming monte-carlo temporal-difference gymnasium https://prathameshsaraf.com/blogs/02-mdps-and-bellman/ https://prathameshsaraf.com/blogs/02-mdps-and-bellman/ Tue, 12 May 2026 00:00:00 GMT How to formalize an environment as a Markov Decision Process, why the value of a state splits into V and Q, and the Bellman equation that ties them together recursively. Derive it once, verify it by hand, then solve it exactly on FrozenLake in Gymnasium. reinforcement-learning mdp bellman-equation gymnasium https://prathameshsaraf.com/blogs/01-rl-intro-and-prerequisites/ https://prathameshsaraf.com/blogs/01-rl-intro-and-prerequisites/ Tue, 05 May 2026 00:00:00 GMT What RL really is, why 'how good' means 'how good on average', and the five pieces of math (expectation, the Markov property, variance, running averages, and discounting) that the rest of the field is built on. Intuition first, one clean derivation each, a worked example, then code with Gymnasium. reinforcement-learning fundamentals probability gymnasium