Autograd Compute Graph

During the forward pass the framework builds a computation graph (DAG) on the fly: every tensor that requires grad records the operation that produced it as grad_fn, and the graph is rebuilt per iteration following the current control flow (dynamic graph). Backprop reverses the graph: topological-sort the DAG and visit it in reverse order from the scalar loss, applying the chain rule $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}$ to push gradients along edges and accumulate them at upstream variables (e.g. for $z = x + y$, $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial y} = 1$).

understanding how any autodiff framework implements backward, hand-writing autograd, and correctly implementing forward/backward for custom layers; interviews compare dynamic vs static graphs.

automatic differentiation removes manual/symbolic gradients — any differentiable program gets its gradient for free. Dynamic graphs (PyTorch eager) are built while executing, naturally support Python control flow, and are easy to debug, at the cost of per-step graph construction; static graphs (TF1 Graph mode) are built once and executed later, enabling global operator-fusion optimizations and smooth deployment, but are inflexible and hard to debug. Pitfalls: in-place ops break backward dependencies, so tensors that require grad forbid in-place modification; leaf tensors decide whether gradients are stored.

Test Your Knowledge: Practice Questions for "Autograd Compute Graph"

Single choice pitfall questions with instant feedback and mistake tracking.