L4MediumPart L · Classical ML & Statistical Simulation
Principal Component Analysis (PCA)
Principal Component Analysis projecting centered features onto top-K SVD right singular vectors.
⏱️ Time:
O(min(N D^2, N^2 D)) SVD 分解开销💾 Space:
O(N * k)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def pca_svd(x: np.ndarray, k: int = 2) -> tuple:
"""
基于 SVD 实现的数值稳定 PCA 降维。
参数:
x: (N, D)
k: 目标降维维度
返回:
x_proj: (N, k) 降维后坐标
components: (D, k) 主成分轴基底
"""
# 1. 均值中心化
mean =
x_centered = x - mean
# 2. 奇异值分解 SVD: X_centered = U * S * Vh
U, S, Vh = np.linalg.svd(x_centered, full_matrices=False)
# Vh 的行向量即为主成分特征向量,提取前 k 个
components = Vh[:k, :].T # (D, k)
# 3. 投影到主成分低维空间
x_proj =
return x_proj, components
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is the covariance between distinct components after PCA projection, and why?
💡
Core Mental Anchor / Mnemonic
Center features around zero, run SVD, and project onto top-K right singular directions.
📐 Mathematical Derivation & Core Formula
### Maximum Variance Projection & SVD Equivalence
For centered data with covariance , maximizing projected variance subject to yields the eigenvalue problem .
Rather than forming explicitly (which squares condition numbers and costs ), taking the SVD directly gives exact principal directions in the top columns of .
For centered data with covariance , maximizing projected variance subject to yields the eigenvalue problem .
Rather than forming explicitly (which squares condition numbers and costs ), taking the SVD directly gives exact principal directions in the top columns of .
🔄 Tensor Dimensions & Shape Flow
x: (N, D) -> subtract mean -> SVD -> top-K V^T rows -> x_centered @ components -> (N, K)
🛡️ Numerical Stability & Pitfalls Checklist
- Always mean-center data (
x - x.mean(axis=0)) before computing SVD or covariance. - Compute principal axes via SVD directly rather than squaring condition numbers in
X^T X.
💻 Reference Code Implementation
import numpy as np
def pca_svd(x: np.ndarray, k: int = 2) -> tuple:
"""
基于 SVD 实现的数值稳定 PCA 降维。
参数:
x: (N, D)
k: 目标降维维度
返回:
x_proj: (N, k) 降维后坐标
components: (D, k) 主成分轴基底
"""
# 1. 均值中心化
mean = np.mean(x, axis=0)
x_centered = x - mean
# 2. 奇异值分解 SVD: X_centered = U * S * Vh
U, S, Vh = np.linalg.svd(x_centered, full_matrices=False)
# Vh 的行向量即为主成分特征向量,提取前 k 个
components = Vh[:k, :].T # (D, k)
# 3. 投影到主成分低维空间
x_proj = x_centered @ components # (N, k)
return x_proj, components
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
# 构造沿对角线强相关的三维数据
x = np.random.randn(50, 1) @ np.array([[1.0, 2.0, 3.0]]) + np.random.randn(50, 3) * 0.01
proj, comp = pca_svd(x, k=1)
assert proj.shape == (50, 1)
assert comp.shape == (3, 1)
print("✓ PCA SVD 降维自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What is the covariance between distinct components after PCA projection, and why?
Zero. Principal eigenvectors are orthogonal and diagonalize the covariance matrix into , eliminating linear correlation across projected dimensions.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.