Autograd Compute Graph

During the forward pass the framework builds a computation graph (DAG) on the fly: every tensor that requires grad records the operation that produced it as grad_fn, and the graph is rebuilt per iteration following the current control flow (dynamic graph). Backprop reverses the graph: topological-sort the DAG and visit it in reverse order from the scalar loss, applying the chain rule $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \frac{\partial y}{\partial x}$ to push gradients along edges and accumulate them at upstream variables (e.g. for $z = x + y$, $\frac{\partial z}{\partial x} = \frac{\partial z}{\partial y} = 1$).

understanding how any autodiff framework implements backward, hand-writing autograd, and correctly implementing forward/backward for custom layers; interviews compare dynamic vs static graphs.

automatic differentiation removes manual/symbolic gradients — any differentiable program gets its gradient for free. Dynamic graphs (PyTorch eager) are built while executing, naturally support Python control flow, and are easy to debug, at the cost of per-step graph construction; static graphs (TF1 Graph mode) are built once and executed later, enabling global operator-fusion optimizations and smooth deployment, but are inflexible and hard to debug. Pitfalls: in-place ops break backward dependencies, so tensors that require grad forbid in-place modification; leaf tensors decide whether gradients are stored.

How does backprop determine the computation order? Why reverse topological order?

Dynamic vs static computation graphs: differences, pros and cons?

Role of grad_fn? How is the chain rule applied layer by layer along the DAG?

Why are in-place operations forbidden or error-prone on tensors that require grad?

Difference between .detach() and requires_grad=False? What is a leaf tensor?