Inference Engineering · Post 03
FlashAttention, one query block at a time
← Prev
1 / 1
Next →
In HBM — the full tensors
4 tokens, head dimension 3, blocks of 2. The 1/√d scale is left off so the scores stay whole numbers.
What happens on this step
On chip — SRAM
Accumulators for the current query block
How O was built this step
Written back to HBM
resident on chip
this step's tile
changed this step
← → arrow keys to step