Matrix Calculus & Softmax Gradient

Matrix calculus and the softmax gradient are whiteboard fundamentals of derivation interviews. First, quadratic forms: expand $x^TAx = \sum_{i,j} x_i A_{ij} x_j$ and differentiate w.r.t. $x_k$ — terms with $i=k$ contribute $\sum_j A_{kj}x_j$, terms with $j=k$ contribute $\sum_i A_{ik}x_i$, so $\frac{\partial (x^TAx)}{\partial x_k} = (Ax)_k + (A^Tx)_k$ and hence $\frac{\partial x^TAx}{\partial x} = (A + A^T)x$ ($= 2Ax$ when $A$ is symmetric). Second, the softmax + cross-entropy joint gradient: with $p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$ and $L = -\sum_i y_i \log p_i$, the softmax derivative splits into two cases — for $i = j$: $\frac{\partial p_i}{\partial z_i} = p_i(1-p_i)$; for $i \neq j$: $\frac{\partial p_j}{\partial z_i} = -p_i p_j$. The chain rule gives $\frac{\partial L}{\partial z_i} = \sum_j \frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i} = -\frac{y_i}{p_i}\cdot p_i(1-p_i) + \sum_{j\neq i}\frac{y_j}{p_j}\cdot p_i p_j = -y_i(1-p_i) + p_i\sum_{j\neq i} y_j$. Since $\sum_j y_j = 1$, we get $-y_i + y_i p_i + p_i - p_i y_i = p_i - y_i$, i.e. $\frac{\partial L}{\partial z_i} = p_i - y_i$.

interviewers ask for on-the-whiteboard derivations of $\frac{\partial L}{\partial z} = p - y$, $\frac{\partial x^TAx}{\partial x} = (A+A^T)x$ and $\frac{\partial}{\partial x}\Vert Ax - b\Vert^2 = 2A^T(Ax-b)$; it is the prerequisite for understanding backpropagation and Jacobian/Hessian dimension checks.

the derivation collapses to a memorizable final form $p_i - y_i$: the two-case signs cancel and the $\log p_j$ in CE exactly cancels the softmax denominator, turning an $O(C^2)$ Jacobian computation into an $O(C)$ vector subtraction — exactly how PyTorch/TensorFlow implement the Softmax+CE gradient layer.

Test Your Knowledge: Practice Questions for "Matrix Calculus & Softmax Gradient"

Single choice pitfall questions with instant feedback and mistake tracking.