FlashAttention

More arithmetic, faster — attention was HBM-bound.

play

naive writes N×N to HBM · Flash keeps tiles in SRAM · same math, fewer trips

N 16HBM trips — · SRAM tiles —

Attention was HBM-bound. Tiles in SRAM → more arithmetic, faster.

Naive attention writes N×N to HBM. Flash keeps Br×Bc tiles in SRAM.