M4-100M4: Sequences & TransformersModel Compression & DistillationEasy
⚡Flashcard Drill➔Mastery:
Three Granularities of Model Pruning: Unstructured, Semi-Structured 2:4, and Structured
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Model pruning removes redundant parameters across three distinct granularities: unstructured pruning maximizes theoretical sparsity but requires sparse kernels, semi-structured 2:4 pruning matches NVIDIA Ampere Sparse Tensor Cores, and structured pruning removes entire channels or heads for immediate hardware acceleration.
📌 Key Takeaways
- •Unstructured pruning: zeroes out individual weights with lowest magnitude; highest compression ratio with zero accuracy drop, but yields zero speedup on standard dense matrix hardware
- •Semi-structured 2:4 sparsity: enforces exactly 2 non-zero values for every 4 consecutive elements; directly accelerated by NVIDIA Ampere/Hopper Sparse Tensor Cores for a hardware speedup
- •Structured pruning: removes complete attention heads, FFN neurons, or transformer layers; directly shrinks dense tensor dimensions, achieving immediate speedup and memory reduction on any hardware
📐 Mathematical Derivations
Mathematical Mechanism: 1. Unstructured Pruning: For weight tensor and binary mask : Non-zero elements are scattered randomly across the matrix. Standard dense hardware (Tensor Cores) cannot skip zero multiplications efficiently without custom sparse matrix formats (CSR/CSC), resulting in slower execution due to memory index lookup overhead. 2. Semi-Structured 2:4 Sparsity (NVIDIA Sparse Tensor Cores): In every group of 4 contiguous values in a row, exactly 2 values must be zero: NVIDIA Ampere architecture compresses the 4 values into 2 FP16 values along with a 2-bit metadata index. The hardware Sparse Tensor Core fetches compressed weights from memory, doubling memory bandwidth efficiency and doubling math execution throughput ( FLOPs). 3. Structured Pruning: Removes entire blocks or dimensions: Reduces dense GEMM matrix dimensions directly, requiring no custom kernels or hardware support.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Accuracy vs Hardware Accelerability Trade-off: - Unstructured: Sparsity with minimal loss, but effective speedup . - Semi-Structured 2:4: Fixed sparsity, guaranteed real-world speedup on A100/H100, requires fine-tuning to recover small accuracy drops. - Structured: Sparsity , guaranteed dense speedup across any CPU/GPU/mobile device, but drops accuracy rapidly beyond pruning. ② Pruning Criteria: Magnitude-based pruning () is simple; second-order Hessian-based pruning (Optimal Brain Damage / SparseGPT) measures loss curvature , preserving accuracy at much higher sparsity levels. ③ Lottery Ticket Hypothesis: Dense randomly-initialized networks contain sparse sub-networks ('winning tickets') that, when trained in isolation from scratch with original initializations, match the accuracy of the full dense model. ④ Pruning LLMs: In 70B+ models, structured pruning of layers or intermediate dimensions often causes severe disruption to emergent capabilities; 2:4 semi-structured pruning with SparseGPT has become the primary production route. ⑤ Interview Strategy: Draw the 2:4 pattern diagram, contrast theoretical sparsity vs physical wall-clock speedup on modern GPUs, and explain how Sparse Tensor Cores achieve hardware acceleration.
⚠️ Common Interview Pitfalls
- ✕Assuming 70% unstructured pruning automatically provides a 70% speedup on GPUs (it usually runs slower due to irregular memory indexing)
- ✕Applying structured pruning aggressively to LLMs without extensive post-training fine-tuning
- ✕Failing to recognize that 2:4 sparsity requires NVIDIA Ampere (A100) or newer GPUs for hardware acceleration
🎯 Interviewer Follow-up Inquiries
- ?How does SparseGPT apply second-order Hessian compensation to prune 100B+ models in hours?
- ?How do NVIDIA Sparse Tensor Cores physically compress 2:4 sparse weights in memory?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚