micrograd — how a neural network
actually learns

00 Start here — what problem is this even solving?

A neural network is just a big math formula with thousands of knobs (calledweights). "Training" means: turn each knob a tiny bit in the direction that makes the network less wrong, over and over, until it's good.

To do that you need one thing: for every knob, which way and how much does turning it change the final error? That number is the gradient. Computing it by hand for thousands of knobs is impossible. micrograd's entire job is to compute all those gradients automatically. That's it. Everything below is how.

01 The only object in the whole engine: Value

micrograd wraps every single number in a little box called a Value. A plain number like 3.0 doesn't remember anything. A Value remembers where it came from — and that memory is what makes automatic gradients possible.

class Value:
    """ stores a single scalar value and its gradient """

    def __init__(self, data, _children=(), _op=''):
        self.data = data            # the actual number, e.g. 3.0
        self.grad = 0               # how much the final answer changes if I change
        self._backward = lambda: None   # a function: how to pass gradient to my inputs
        self._prev = set(_children)     # the Values I was built from
        self._op = _op                  # which op made me ('+', '*', ...) — for debugging

In plain words, each Value carries:

• data — the actual number.
• grad — the gradient, starts at 0, gets filled in later. This is the answer we're after.
• _prev — the other Values this one was built from (e.g. c = a + b → c remembers a and b).
• _backward — a little function that passes gradient backward to _prev. Empty for now; each operation fills it in.

So a Value is really a node in a graph: numbers flow forward to build it, and gradient will flow backward through it.

02 How an operation secretly records itself

Here's the clever part. When you multiply two Values, micrograd doesn't just compute the answer — it also writes down how to reverse it later. Watch:

def __mul__(self, other):                       # runs when you write  a * b
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')

    def _backward():
        self.grad  += other.data * out.grad
        other.grad += self.data  * out.grad
    out._backward = _backward

    return out

Three things happen the moment you write a * b:

• Compute forward — out.data = a.data * b.data. The normal multiplication.
• Remember the parents — out stores (self, other), i.e. a and b.
• Stash the reverse rule — _backward is a small function saved for later. It holds the calculus fact that for multiplication, each input's gradient is the otherinput (times the gradient arriving from above, out.grad).

Addition is even simpler — a + passes gradient straight through to both inputs, unchanged:

def __add__(self, other):                       # runs when you write  a + b
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')

    def _backward():
        self.grad  += out.grad
        other.grad += out.grad
    out._backward = _backward

    return out

Every operation in the file follows this identical shape — only the reverse rule differs:

Those four are the calculus you covered in Module 0.2 (the chain rule). micrograd just stores each little rule so a computer can chain them.

03 The heart: backward() runs the chain rule for you

After you've built an expression, calling .backward() on the final answer fills in.grad for every Value that led to it. Here's the whole method:

def backward(self):

    # 1) put every node in order: children always before their parents
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)

    # 2) seed the very last value, then sweep backward through the graph
    self.grad = 1
    for v in reversed(topo):
        v._backward()

It does two things:

Step 1 — order the nodes. You can't compute a node's gradient until everything that depends on it is done. build_topo lists each node after its children, then we go through that list in reverse — so we always handle a node before the nodes that feed it.

Step 2 — seed and sweep. The final value's gradient with respect to itself is 1(nudging the answer changes the answer one-for-one). Set that, then call each saved_backward() in order. Each call multiplies its local rule by the gradient already arrived from above. That cascade of multiplications IS the chain rule, executing itself across the whole graph.

04 Watch it happen — one worked example, slowly

Take a tiny expression and compute every gradient by hand, exactly as the code does:

a = 2
b = -3
c = 10

e = a * b        #  e = -6
d = e + c        #  d =  4
L = d * (-2)     #  L = -8   ← the final answer

Now L.backward(). Start the final value at 1, then walk backward:

Result: ∂L/∂a = 6, ∂L/∂b = −4. We computed the gradient of the final answer with respect to the inputs — using no calculus of our own, just the little saved rules multiplied along the chain. That is the entire trick, and it scales to billions of nodes.

05 The one subtle line: +=, not =

Every reverse rule adds to .grad (+=). This matters when a value is used in more than one place. If x feeds two parts of the formula, gradient comes back from both paths, and the correct total is their sum. += accumulates them; a plain = would keep only the last and silently give wrong answers.

06 A network is just Values wired together

nn.py (60 lines) builds neurons out of Values. A neuron computesweights · inputs + bias, then optionally a ReLU:

class Neuron(Module):

    def __init__(self, nin, nonlin=True):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]  # a weight per input
        self.b = Value(0)                                           # one bias
        self.nonlin = nonlin

    def __call__(self, x):
        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)      # w·x + b
        return act.relu() if self.nonlin else act

The magic: every wi*xi and + here calls the operations from step 02 — sosimply evaluating the network automatically builds the full gradient graph. There is no separate "write the backward pass" step; it was recorded as you computed forward. Stack neurons intoLayers, stack layers into an MLP, and you have a real (tiny) neural network — still just Values underneath.

07 Training, in full — the whole loop

Now everything connects. This is how every model on earth trains, micrograd included:

for k in range(epochs):
    out  = model(x)                     # forward: also builds the autograd graph
    loss = (out - target)**2            # a Value measuring how wrong we are

    model.zero_grad()                   # wipe last step's gradients (the += reason)
    loss.backward()                     # fill every parameter's .grad automatically

    for p in model.parameters():
        p.data -= learning_rate * p.grad   # gradient descent — nudge toward less loss

1. Forward — run the network; this also builds the graph.
2. Loss — one Value measuring how wrong the output is.
3. zero_grad — clear last step's accumulated gradients.
4. backward — fill every knob's .grad automatically (steps 03–04).
5. step — nudge each knob down its gradient: p.data -= lr * p.grad.

That last line is literally gradient descent: θ ← θ − η·∇L. Repeat a few thousand times → the network learns. A 70-billion-parameter model runs this exact loop — just with arrays instead of scalars, on a GPU instead of in Python.

08 Is it actually correct? Yes — they check against PyTorch

The repo's test builds the same messy expression in both micrograd and PyTorch, runsbackward() on each, and asserts the gradients match. Our 154 lines produce the identical answer the industrial framework does:

x = Value(-4.0)                  # micrograd
z = 2 * x + 2 + x
q = z.relu() + z * x
y = h + q + q * x
y.backward()                     # our gradient: x.grad

x = torch.Tensor([-4.0]); x.requires_grad = True   # PyTorch, same math
# ... identical expression ...
y.backward()

assert xmg.grad == xpt.grad.item()   # ✅ micrograd's gradient == PyTorch's

That's the whole point of reading it: micrograd isn't a simplified cartoon — it's the real mechanism, small enough to hold in your head, verified against the real thing.