Backprop is just the chain rule with bookkeeping

Backprop is not a special algorithm. It is the chain rule, applied in an order that avoids recomputing anything. The only reason it needs code is that something has to remember the order.

The one idea

Nothing deeper happens in a hundred-layer network. When a value feeds three places, its gradient is the sum of what comes back from each.

The one data structure

Build the graph as a side effect of the forward pass. Every operation records its parents and a closure holding the local derivative.

class Value:
    def __init__(self, data, parents=(), backward=lambda: None):
        self.data = data
        self.grad = 0.0
        self._parents = parents
        self._backward = backward

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))

        def backward():
            self.grad  += other.data * out.grad
            other.grad += self.data  * out.grad

        out._backward = backward
        return out

Note the +=. That is the fan-out rule, and the most common bug in a hand-rolled engine.

graph LR
x[x] --> m((*))
w[w] --> m
m --> a((+))
b[b] --> a
a --> t[tanh]
t --> L[loss]

Ordering is the algorithm

A gradient is final only once every consumer has contributed. Sort topologically, reverse, call each closure once.

def backward(self):
    order, seen = [], set()

    def build(v):
        if v in seen:
            return
        seen.add(v)
        for p in v._parents:
            build(p)
        order.append(v)

    build(self)
    self.grad = 1.0
    for node in reversed(order):
        node._backward()

That is the entire engine. PyTorch is this plus tensors, kernels, and thirty thousand lines of making it fast.

Check it