SVD & Eigen-Decomposition

For any $A \in \mathbb{R}^{m \times n}$, the singular value decomposition is $A = U\Sigma V^T$, where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal and $\Sigma$ is diagonal with singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_{\min(m,n)} \ge 0$ (zeros allowed). Geometrically the map is "rotate + scale + rotate": $V^T$ rotates the input onto the coordinate axes, $\Sigma$ scales each axis by $\sigma_i$, and $U$ rotates into the output space — the unit sphere becomes a hyper-ellipse whose semi-axis lengths are the $\sigma_i$, the longest axis $u_1$ being the most stretched direction. The singular values satisfy $\sigma_i^2 = \lambda_i(A^TA)$; eigenvectors of $A^TA$ form $V$, eigenvectors of $AA^T$ form $U$. EVD: a symmetric matrix decomposes as $A = Q\Lambda Q^T$ ($Q$ orthogonal) with real eigenvalues and singular values $= |\lambda_i|$. The condition number $\kappa = \sigma_1 / \sigma_n$ measures numerical stability: larger $\kappa$ means more ill-conditioned, and relative errors in the solution can be amplified by up to $\kappa$.

detecting ill-conditioned problems (equation solving, gradient-descent convergence); numerical implementation of PCA and dimensionality reduction; low-rank approximation and compression; the pseudoinverse $A^+ = V\Sigma^+U^T$; image denoising, whitening, and matrix factorization in recommenders.

EVD only exists for diagonalizable square matrices, and non-symmetric matrices can have complex, numerically unstable eigenvalues; SVD exists for every matrix (rectangular or rank-deficient) with unique singular values, making it the most stable decomposition in numerical linear algebra. The condition number $\kappa = \sigma_1/\sigma_n$ directly quantifies how much input noise is amplified — the yardstick for whether a linear system is trustworthy and how fast first-order optimization will converge.

Test Your Knowledge: Practice Questions for "SVD & Eigen-Decomposition"

Single choice pitfall questions with instant feedback and mistake tracking.