Matrix Calculus & Softmax Gradient

Matrix calculus and the softmax gradient are whiteboard fundamentals of derivation interviews. First, quadratic forms: expand $x^TAx = \sum_{i,j} x_i A_{ij} x_j$ and differentiate w.r.t. $x_k$ — terms with $i=k$ contribute $\sum_j A_{kj}x_j$, terms with $j=k$ contribute $\sum_i A_{ik}x_i$, so $\frac{\partial (x^TAx)}{\partial x_k} = (Ax)_k + (A^Tx)_k$ and hence $\frac{\partial x^TAx}{\partial x} = (A + A^T)x$ ($= 2Ax$ when $A$ is symmetric). Second, the softmax + cross-entropy joint gradient: with $p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$ and $L = -\sum_i y_i \log p_i$, the softmax derivative splits into two cases — for $i = j$: $\frac{\partial p_i}{\partial z_i} = p_i(1-p_i)$; for $i \neq j$: $\frac{\partial p_j}{\partial z_i} = -p_i p_j$. The chain rule gives $\frac{\partial L}{\partial z_i} = \sum_j \frac{\partial L}{\partial p_j}\frac{\partial p_j}{\partial z_i} = -\frac{y_i}{p_i}\cdot p_i(1-p_i) + \sum_{j\neq i}\frac{y_j}{p_j}\cdot p_i p_j = -y_i(1-p_i) + p_i\sum_{j\neq i} y_j$. Since $\sum_j y_j = 1$, we get $-y_i + y_i p_i + p_i - p_i y_i = p_i - y_i$, i.e. $\frac{\partial L}{\partial z_i} = p_i - y_i$.

interviewers ask for on-the-whiteboard derivations of $\frac{\partial L}{\partial z} = p - y$, $\frac{\partial x^TAx}{\partial x} = (A+A^T)x$ and $\frac{\partial}{\partial x}\Vert Ax - b\Vert^2 = 2A^T(Ax-b)$; it is the prerequisite for understanding backpropagation and Jacobian/Hessian dimension checks.

the derivation collapses to a memorizable final form $p_i - y_i$: the two-case signs cancel and the $\log p_j$ in CE exactly cancels the softmax denominator, turning an $O(C^2)$ Jacobian computation into an $O(C)$ vector subtraction — exactly how PyTorch/TensorFlow implement the Softmax+CE gradient layer.

Whiteboard: expand $x^TAx = \sum_{i,j}x_iA_{ij}x_j$ to derive $\frac{\partial x^TAx}{\partial x} = (A+A^T)x$; why does it reduce to $2Ax$ when $A$ is symmetric?

Whiteboard the softmax+CE joint gradient $\frac{\partial L}{\partial z_i} = p_i - y_i$: derive $\frac{\partial p_j}{\partial z_i}$ for $i=j$ vs $i\neq j$ and simplify with $\sum_j y_j = 1$.

Derive the least-squares gradient $\frac{\partial}{\partial x}\Vert Ax - b\Vert_2^2 = 2A^T(Ax-b)$ and apply the dimension-check rule to each factor.

Why does $\log p_j$ in CE cancel the softmax denominator to leave $p-y$? Is the gradient still clean under squared loss $\frac{1}{2}(p_i - y_i)^2$?

Dimension-check the chain rule: given $\frac{\partial L}{\partial z} \in \mathbb{R}^C$ and the shape of $\frac{\partial z}{\partial W}$, how do they multiply to yield $\frac{\partial L}{\partial W}$?