M4-100M4: Sequences & TransformersModel Compression & DistillationEasy
⚡Flashcard Drill➔Mastery:

Three Granularities of Model Pruning: Unstructured, Semi-Structured 2:4, and Structured

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-101 →
📐 Mathematical Definition
unstructured: {wi=0};structured: prune channels/heads/layers;semi: 2:4\text{unstructured}:\ \{w_i=0\};\qquad \text{structured}:\ \text{prune channels/heads/layers};\qquad \text{semi}:\ 2{:}4
⚡ One-Sentence Core Answer
Model pruning removes redundant parameters across three distinct granularities: unstructured pruning maximizes theoretical sparsity but requires sparse kernels, semi-structured 2:4 pruning matches NVIDIA Ampere Sparse Tensor Cores, and structured pruning removes entire channels or heads for immediate hardware acceleration.

📌 Key Takeaways

  • •
    Unstructured pruning: zeroes out individual weights with lowest magnitude; highest compression ratio with zero accuracy drop, but yields zero speedup on standard dense matrix hardware
  • •
    Semi-structured 2:4 sparsity: enforces exactly 2 non-zero values for every 4 consecutive elements; directly accelerated by NVIDIA Ampere/Hopper Sparse Tensor Cores for a hardware 2×2\times speedup
  • •
    Structured pruning: removes complete attention heads, FFN neurons, or transformer layers; directly shrinks dense tensor dimensions, achieving immediate speedup and memory reduction on any hardware

📐 Mathematical Derivations

Mathematical Mechanism: 1. Unstructured Pruning: For weight tensor W∈Rd1×d2W \in \mathbb{R}^{d_1 \times d_2} and binary mask M∈{0,1}d1×d2M \in \{0, 1\}^{d_1 \times d_2}: Wsparse=W⊙M,Mij=I(∣Wij∣≥τ)W_{\text{sparse}} = W \odot M, \quad M_{ij} = \mathbb{I}(|W_{ij}| \ge \tau) Non-zero elements are scattered randomly across the matrix. Standard dense hardware (Tensor Cores) cannot skip zero multiplications efficiently without custom sparse matrix formats (CSR/CSC), resulting in slower execution due to memory index lookup overhead. 2. Semi-Structured 2:4 Sparsity (NVIDIA Sparse Tensor Cores): In every group of 4 contiguous values in a row, exactly 2 values must be zero: ∀k:∑i=4k4k+3Mi=2\forall k: \sum_{i=4k}^{4k+3} M_{i} = 2 NVIDIA Ampere architecture compresses the 4 values into 2 FP16 values along with a 2-bit metadata index. The hardware Sparse Tensor Core fetches compressed weights from memory, doubling memory bandwidth efficiency and doubling math execution throughput (2×2\times FLOPs). 3. Structured Pruning: Removes entire blocks or dimensions: Wpruned=W[:d1′,:d2′]∈Rd1′×d2′(d1′<d1,d2′<d2)W_{\text{pruned}} = W_{[:d_1', :d_2']} \in \mathbb{R}^{d_1' \times d_2'} \quad (d_1' < d_1, d_2' < d_2) Reduces dense GEMM matrix dimensions directly, requiring no custom kernels or hardware support.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Accuracy vs Hardware Accelerability Trade-off: - Unstructured: Sparsity 80–90%80\text{--}90\% with minimal loss, but effective speedup ≈1.0×\approx 1.0\times. - Semi-Structured 2:4: Fixed 50%50\% sparsity, guaranteed 1.5–1.8×1.5\text{--}1.8\times real-world speedup on A100/H100, requires fine-tuning to recover small accuracy drops. - Structured: Sparsity 20–40%20\text{--}40\%, guaranteed dense speedup across any CPU/GPU/mobile device, but drops accuracy rapidly beyond 30%30\% pruning. ② Pruning Criteria: Magnitude-based pruning (∣W∣|W|) is simple; second-order Hessian-based pruning (Optimal Brain Damage / SparseGPT) measures loss curvature Δw=−H−1w[H−1]ii\Delta w = -\frac{H^{-1} w}{[H^{-1}]_{ii}}, preserving accuracy at much higher sparsity levels. ③ Lottery Ticket Hypothesis: Dense randomly-initialized networks contain sparse sub-networks ('winning tickets') that, when trained in isolation from scratch with original initializations, match the accuracy of the full dense model. ④ Pruning LLMs: In 70B+ models, structured pruning of layers or intermediate dimensions often causes severe disruption to emergent capabilities; 2:4 semi-structured pruning with SparseGPT has become the primary production route. ⑤ Interview Strategy: Draw the 2:4 pattern diagram, contrast theoretical sparsity vs physical wall-clock speedup on modern GPUs, and explain how Sparse Tensor Cores achieve hardware acceleration.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming 70% unstructured pruning automatically provides a 70% speedup on GPUs (it usually runs slower due to irregular memory indexing)
  • ✕
    Applying structured pruning aggressively to LLMs without extensive post-training fine-tuning
  • ✕
    Failing to recognize that 2:4 sparsity requires NVIDIA Ampere (A100) or newer GPUs for hardware acceleration
🎯 Interviewer Follow-up Inquiries
  • ?
    How does SparseGPT apply second-order Hessian compensation to prune 100B+ models in hours?
  • ?
    How do NVIDIA Sparse Tensor Cores physically compress 2:4 sparse weights in memory?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-099: Knowledge Distillation Framework and the Role of Temperature $T$📋 Back to BankNext →M4-101: On-Policy vs. Off-Policy Data in Knowledge Distillation for LLMs