M6-094M6: Multimodal & Generative ModelsVideo, 3D & Audio Generative ModelsMedium
⚡Flashcard Drill➔Mastery:

Neural Radiance Fields (NeRF) vs 3D Gaussian Splatting: Continuous Volumetric vs Discrete Primitives

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M6-095 →
📐 Mathematical Definition
NeRF: Fθ(x,d)→(c,σ);3DGS: {(μi,Σi,ci,αi)} rasterized\text{NeRF}:\ F_\theta(x,d)\to(c,\sigma);\qquad \text{3DGS}:\ \{(\mu_i,\Sigma_i,c_i,\alpha_i)\}\ \text{rasterized}
⚡ One-Sentence Core Answer
NeRF models 3D scenes as continuous implicit neural volumetric functions optimized via ray marching, while 3D Gaussian Splatting represents scenes using explicit anisotropic 3D Gaussians rendered via fast tile-based rasterization.

📌 Key Takeaways

  • •
    NeRF representation (Mildenhall et al.): encodes continuous scene geometry and view-dependent color using a coordinate MLP  Fθ(x,d)→(σ,c) \,F_\theta(x, d) \to (\sigma, c)\,, rendering via numerical volume integration along camera rays
  • •
    3D Gaussian Splatting (Kerbl et al.): models scenes using millions of explicit anisotropic 3D Gaussians defined by position, covariance matrix, opacity, and spherical harmonics
  • •
    Rendering speed breakthrough: NeRF requires evaluating hundreds of MLP forward passes per pixel (seconds per frame), while 3DGS achieves real-time 100+ 100+\, FPS rendering via GPU tile-based rasterization

📐 Mathematical Derivations

Mathematical Mechanism: 1. NeRF Volumetric Ray Marching: For camera ray r(t)=o+tdr(t) = o + t d, rendered pixel color C(r)C(r) integrates continuous volume density σ(r(t))\sigma(r(t)) and color c(r(t),d)c(r(t), d) between near bound tnt_n and far bound tft_f: C(r)=∫tntfT(t)σ(r(t))c(r(t),d)dt,T(t)=exp⁡(−∫tntσ(r(s))ds)C(r) = \int_{t_n}^{t_f} T(t) \sigma(r(t)) c(r(t), d) dt, \quad T(t) = \exp\left( -\int_{t_n}^t \sigma(r(s)) ds \right) Approximated numerically via stratified quadrature sampling across KK discrete points: C^(r)=∑i=1KTi(1−exp⁡(−σiδi))ci,Ti=∏j=1i−1exp⁡(−σjδj)\hat{C}(r) = \sum_{i=1}^K T_i (1 - \exp(-\sigma_i \delta_i)) c_i, \quad T_i = \prod_{j=1}^{i-1} \exp(-\sigma_j \delta_j) 2. 3D Gaussian Splatting Formulation: A 3D Gaussian is parameterized by mean position μ∈R3\mu \in \mathbb{R}^3 and 3D covariance matrix Σ∈R3×3\Sigma \in \mathbb{R}^{3 \times 3}: G(x)=exp⁡(−12(x−μ)TΣ−1(x−μ))G(x) = \exp\left( -\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu) \right) To ensure positive semi-definiteness during gradient descent, covariance is decomposed into scaling vector s∈R3s \in \mathbb{R}^3 and rotation quaternion q∈R4q \in \mathbb{R}^4: Σ=R(q)S(s)S(s)TR(q)T\Sigma = R(q) S(s) S(s)^T R(q)^T 3. 2D Projective Splatting (Zwicker et al.): Given viewing transformation WW and projective Jacobian JJ, the 2D projected screen covariance is: Σ′=JWΣWTJT∈R2×2\Sigma' = J W \Sigma W^T J^T \in \mathbb{R}^{2 \times 2} Pixels are colored via tile-based alpha-blending of sorted Gaussians: C=∑i∈Nciαi∏j=1i−1(1−αj),αi=oi⋅G2D(p−μi′)C = \sum_{i \in \mathcal{N}} c_i \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j), \quad \alpha_i = o_i \cdot G_{2\text{D}}(p - \mu'_i)

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Paradigm Shift from Implicit to Explicit: NeRF was revolutionary in 2020 for eliminating discrete polygonal meshes and achieving photorealistic novel view synthesis. However, querying a deep MLP 128 times per ray for a 1920×10801920 \times 1080 frame (2×1062 \times 10^6 rays) requires hundreds of millions of neural network evaluations per image, capping rendering speed at 0.1-1 FPS. 3D Gaussian Splatting eliminates neural networks at inference entirely: rendering is pure GPU rasterization (sorting 2D Gaussians and executing hardware alpha blending), reaching 100-200 FPS on consumer hardware at equal or higher visual quality. ② Memory Footprint and Storage: (a) NeRF: Compact representation; the entire scene is compressed into a 5–50 MB5\text{--}50\,\text{MB} MLP weight checkpoint. (b) 3DGS: Storing 2-5 million 3D Gaussians (each storing position, rotation, scale, opacity, and 16 spherical harmonic coefficients) consumes 500 MB500\,\text{MB} to 1.5 GB1.5\,\text{GB} of RAM/disk storage. Compression techniques (vector quantization of spherical harmonics, Gaussian pruning) reduce footprint by 10×10\times. ③ Adaptive Density Control in 3DGS: 3DGS periodically splits large Gaussians in under-reconstructed regions and clones small Gaussians in high-frequency detail regions, while pruning transparent Gaussians (oi<ϵo_i < \epsilon). ⑤ Interview Strategy: Contrast implicit volumetric MLP rendering against explicit Gaussian primitive rasterization, write NeRF's volume rendering quadrature equation, derive 3DGS covariance projection Σ′=JWΣWTJT\Sigma' = J W \Sigma W^T J^T, and compare rendering speed vs memory footprint.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting real-time interactive rendering with standard vanilla NeRF; vanilla NeRF requires seconds to render a single frame
  • ✕
    Optimizing covariance matrix Σ\Sigma directly without scale-rotation decomposition (RSSTRTR S S^T R^T), causing non-positive-definite covariance collapse
  • ✕
    Underestimating the storage footprint of raw uncompressed 3D Gaussian splat files (1GB+1\text{GB}+) in web deployment pipelines
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the scale-rotation decomposition Σ=RSSTRT\Sigma = R S S^T R^T mathematically necessary when optimizing 3D Gaussian Splatting?
  • ?
    How does tile-based GPU rasterization enable 3D Gaussian Splatting to render complex scenes at over 100 FPS?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM6-093: Spatio-Temporal Attention Architectures in Video Diffusion Models📋 Back to BankNext →M6-095: Foundational Pipelines of Speech Recognition (ASR) and Text-to-Speech (TTS)