Bytes become ids

BPE is compression. English vs Go vs YAML.

play

BPE · chars start as tokens · merge the hottest pair · compression is not meaning

vocab — · next — · bytes/tok —

English vs Go vs YAML. Bytes per token is the fair meter.

The model never sees characters. Bad compression = more KV = more money.