Jacobian & Hessian

For $f: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian is the $m \times n$ matrix $J$ with entries $J_{ij} = \partial f_i / \partial x_j$ — each row is the gradient row-vector of component $f_i$. Dimension-check rule: $n$-dimensional input and $m$-dimensional output imply a Jacobian of shape $m \times n$ (never $n \times m$); for a scalar function ($m=1$) the Jacobian is the transpose of the gradient $\nabla f$. It provides the first-order linear approximation $f(x + \delta) \approx f(x) + J(x)\delta$. For the quadratic form $f(x) = x^TAx$, the gradient is $\nabla f = (A + A^T)x$, which becomes $\nabla f = 2Ax$ when $A$ is symmetric. Hessian: for a scalar $f: \mathbb{R}^n \to \mathbb{R}$, the symmetric matrix of second partials $H_{ij} = \partial^2 f / \partial x_i \partial x_j \in \mathbb{R}^{n \times n}$. Convexity criterion: if $f$ is twice continuously differentiable, $f$ is convex iff $H(x) \succeq 0$ (positive semidefinite) for all $x$, and strictly convex iff $H(x) \succ 0$. Chain rule: for $h = g \circ f$, $J_h = J_g \cdot J_f$, and the shape product $(p \times m)(m \times n) = p \times n$ doubles as a consistency check.

dimension checking while hand-deriving backprop gradients; Newton's method $x_{t+1} = x_t - H^{-1}\nabla f$ (second-order curvature); convexity analysis (KL divergence, cross-entropy, quadratic losses); second-order optimization, Fisher information, Laplace approximation.

the most common source of error in matrix-calculus derivations is transposes and shapes — "the Jacobian is always $m \times n$" plus chain-rule shape multiplication gives a uniqueness check for any derivation; Hessian positive definiteness operationalizes convexity: convex ⟹ local optimum = global optimum, the fundamental criterion for whether a learning objective is tractable; Newton's method exploits curvature $H^{-1}$ to converge in one step on ill-conditioned problems where first-order methods crawl along narrow valleys.

Test Your Knowledge: Practice Questions for "Jacobian & Hessian"

Single choice pitfall questions with instant feedback and mistake tracking.