Softmax and cross-entropy

Logits → distribution. Loss is surprise at the true id.

play

logits → softmax → cross-entropy · drag the z sliders, click a bar to set the true class

z₀ z₁ z₂ z₃

true class0

p(true)—

cross-entropy—

Drag logits. Click the true class. −log p is pretraining.

L = −log softmax(z)_{y}

Softmax is not optional. −log p(true) is pretraining.