SVD & Eigen-Decomposition

For any $A \in \mathbb{R}^{m \times n}$, the singular value decomposition is $A = U\Sigma V^T$, where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal and $\Sigma$ is diagonal with singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_{\min(m,n)} \ge 0$ (zeros allowed). Geometrically the map is "rotate + scale + rotate": $V^T$ rotates the input onto the coordinate axes, $\Sigma$ scales each axis by $\sigma_i$, and $U$ rotates into the output space — the unit sphere becomes a hyper-ellipse whose semi-axis lengths are the $\sigma_i$, the longest axis $u_1$ being the most stretched direction. The singular values satisfy $\sigma_i^2 = \lambda_i(A^TA)$; eigenvectors of $A^TA$ form $V$, eigenvectors of $AA^T$ form $U$. EVD: a symmetric matrix decomposes as $A = Q\Lambda Q^T$ ($Q$ orthogonal) with real eigenvalues and singular values $= |\lambda_i|$. The condition number $\kappa = \sigma_1 / \sigma_n$ measures numerical stability: larger $\kappa$ means more ill-conditioned, and relative errors in the solution can be amplified by up to $\kappa$.

detecting ill-conditioned problems (equation solving, gradient-descent convergence); numerical implementation of PCA and dimensionality reduction; low-rank approximation and compression; the pseudoinverse $A^+ = V\Sigma^+U^T$; image denoising, whitening, and matrix factorization in recommenders.

EVD only exists for diagonalizable square matrices, and non-symmetric matrices can have complex, numerically unstable eigenvalues; SVD exists for every matrix (rectangular or rank-deficient) with unique singular values, making it the most stable decomposition in numerical linear algebra. The condition number $\kappa = \sigma_1/\sigma_n$ directly quantifies how much input noise is amplified — the yardstick for whether a linear system is trustworthy and how fast first-order optimization will converge.

Compute the SVD of a 2×2 matrix by hand: eigenvalues/eigenvectors of $A^TA$ give $\sigma_i$ and $V$, then $u_i = Av_i/\sigma_i$ gives $U$; verify $A = U\Sigma V^T$.

Derive $\sigma_i^2 = \lambda_i(A^TA)$; why does SVD exist for every matrix while EVD needs a diagonalizable square matrix?

Geometry: why does the unit circle map to an ellipse with semi-axes $\sigma_i$? How do $V^T$, $\Sigma$, $U$ correspond to rotate, scale, rotate?

How does the condition number $\kappa = \sigma_1/\sigma_n$ affect numerical stability? How much can ill-conditioned systems (e.g. Hilbert matrices) amplify errors?

How do SVD and EVD relate for symmetric matrices? How can SVD directly read off bases of the four fundamental subspaces (first $r$ columns of $V$ = row space, etc.)?