SGD, momentum, Adam

Same saddle. Three paths. β→0 makes momentum = SGD.

play

same start on a saddle · SGD stalls · momentum coasts · Adam still steps

η 0.08β 0.90SGD mom Adam

Same saddle. SGD crawls. Momentum coasts. Adam still steps.

Adam divides by the gradient’s own size. Tiny slope still steps.