L4MediumPart L · Classical ML & Statistical Simulation

Principal Component Analysis (PCA)

⚡Flashcard Drill➔

Principal Component Analysis projecting centered features onto top-K SVD right singular vectors.

⏱️ Time: O(min(N D^2, N^2 D)) SVD 分解开销
💾 Space: O(N * k)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def pca_svd(x: np.ndarray, k: int = 2) -> tuple:
"""
基于 SVD 实现的数值稳定 PCA 降维。
参数:
x: (N, D)
k: 目标降维维度
返回:
x_proj: (N, k) 降维后坐标
components: (D, k) 主成分轴基底
"""
# 1. 均值中心化
mean =
x_centered = x - mean
# 2. 奇异值分解 SVD: X_centered = U * S * Vh
U, S, Vh = np.linalg.svd(x_centered, full_matrices=False)
# Vh 的行向量即为主成分特征向量,提取前 k 个
components = Vh[:k, :].T # (D, k)
# 3. 投影到主成分低维空间
x_proj =
return x_proj, components
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is the covariance between distinct components after PCA projection, and why?
Next: Bootstrap Resampling & 95% Confidence Interval →
💡

Core Mental Anchor / Mnemonic

Center features around zero, run SVD, and project onto top-K right singular directions.

📐 Mathematical Derivation & Core Formula

Σ=1NX⊤X=VΛV⊤,Xproj=XV:,:k\Sigma = \frac{1}{N} X^\top X = V \Lambda V^\top, \quad X_{\text{proj}} = X V_{:, :k}
### Maximum Variance Projection & SVD Equivalence
For centered data X∈RN×DX \in \mathbb{R}^{N \times D} with covariance Σ=1NX⊤X\Sigma = \frac{1}{N} X^\top X, maximizing projected variance max⁡uu⊤Σu\max_u u^\top \Sigma u subject to ∥u∥2=1\|u\|_2 = 1 yields the eigenvalue problem Σu=λu\Sigma u = \lambda u.
Rather than forming X⊤XX^\top X explicitly (which squares condition numbers and costs O(ND2)O(ND^2)), taking the SVD X=USV⊤X = U S V^\top directly gives exact principal directions in the top KK columns of VV.

🔄 Tensor Dimensions & Shape Flow

x: (N, D) -> subtract mean -> SVD -> top-K V^T rows -> x_centered @ components -> (N, K)

🛡️ Numerical Stability & Pitfalls Checklist

  • Always mean-center data (x - x.mean(axis=0)) before computing SVD or covariance.
  • Compute principal axes via SVD directly rather than squaring condition numbers in X^T X.

💻 Reference Code Implementation

import numpy as np

def pca_svd(x: np.ndarray, k: int = 2) -> tuple:
    """
    基于 SVD 实现的数值稳定 PCA 降维。
    参数:
        x: (N, D)
        k: 目标降维维度
    返回:
        x_proj: (N, k) 降维后坐标
        components: (D, k) 主成分轴基底
    """
    # 1. 均值中心化
    mean = np.mean(x, axis=0)
    x_centered = x - mean
    
    # 2. 奇异值分解 SVD: X_centered = U * S * Vh
    U, S, Vh = np.linalg.svd(x_centered, full_matrices=False)
    
    # Vh 的行向量即为主成分特征向量,提取前 k 个
    components = Vh[:k, :].T  # (D, k)
    
    # 3. 投影到主成分低维空间
    x_proj = x_centered @ components # (N, k)
    
    return x_proj, components

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
# 构造沿对角线强相关的三维数据
x = np.random.randn(50, 1) @ np.array([[1.0, 2.0, 3.0]]) + np.random.randn(50, 3) * 0.01
proj, comp = pca_svd(x, k=1)
assert proj.shape == (50, 1)
assert comp.shape == (3, 1)
print("✓ PCA SVD 降维自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What is the covariance between distinct components after PCA projection, and why?
Zero. Principal eigenvectors are orthogonal and diagonalize the covariance matrix into Λ=diag(λ1,…,λk)\Lambda = \mathrm{diag}(\lambda_1, \dots, \lambda_k), eliminating linear correlation across projected dimensions.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Logistic Regression with L2 RegularizationBack to Part L & All 105Next: Bootstrap Resampling & 95% Confidence Interval →