Jacobian & Hessian

For $f: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian is the $m \times n$ matrix $J$ with entries $J_{ij} = \partial f_i / \partial x_j$ — each row is the gradient row-vector of component $f_i$. Dimension-check rule: $n$-dimensional input and $m$-dimensional output imply a Jacobian of shape $m \times n$ (never $n \times m$); for a scalar function ($m=1$) the Jacobian is the transpose of the gradient $\nabla f$. It provides the first-order linear approximation $f(x + \delta) \approx f(x) + J(x)\delta$. For the quadratic form $f(x) = x^TAx$, the gradient is $\nabla f = (A + A^T)x$, which becomes $\nabla f = 2Ax$ when $A$ is symmetric. Hessian: for a scalar $f: \mathbb{R}^n \to \mathbb{R}$, the symmetric matrix of second partials $H_{ij} = \partial^2 f / \partial x_i \partial x_j \in \mathbb{R}^{n \times n}$. Convexity criterion: if $f$ is twice continuously differentiable, $f$ is convex iff $H(x) \succeq 0$ (positive semidefinite) for all $x$, and strictly convex iff $H(x) \succ 0$. Chain rule: for $h = g \circ f$, $J_h = J_g \cdot J_f$, and the shape product $(p \times m)(m \times n) = p \times n$ doubles as a consistency check.

dimension checking while hand-deriving backprop gradients; Newton's method $x_{t+1} = x_t - H^{-1}\nabla f$ (second-order curvature); convexity analysis (KL divergence, cross-entropy, quadratic losses); second-order optimization, Fisher information, Laplace approximation.

the most common source of error in matrix-calculus derivations is transposes and shapes — "the Jacobian is always $m \times n$" plus chain-rule shape multiplication gives a uniqueness check for any derivation; Hessian positive definiteness operationalizes convexity: convex ⟹ local optimum = global optimum, the fundamental criterion for whether a learning objective is tractable; Newton's method exploits curvature $H^{-1}$ to converge in one step on ill-conditioned problems where first-order methods crawl along narrow valleys.

Dimension check: why is the Jacobian of $f: \mathbb{R}^n \to \mathbb{R}^m$ $m \times n$ and not $n \times m$? Write the Jacobian of $f(x,y) = (x^2 + y, xy)$ by hand.

Derive: gradient $(A + A^T)x$ and Hessian $A + A^T$ of the quadratic form $f(x) = x^TAx$; why does it simplify to $2Ax$ when $A$ is symmetric?

By hand: compute the Jacobian/Hessian of $f(x_1, x_2) = x_1^2 + 3x_1x_2 + 2x_2^2$ and test convexity via the principal-minor criterion ($a > 0$ and $ac - b^2 > 0$) or eigenvalues.

Prove: a twice-differentiable $f$ is convex iff its Hessian is everywhere positive semidefinite; why does this matter for global optimality (convex ⟹ local = global)?

Chain-rule dimension check: for the scalar loss $L = \Vert Xw - y \Vert^2$, use the dimension rule to confirm the gradient is $2X^T(Xw - y)$ and not any other transpose arrangement.