M6-094M6: Multimodal & Generative ModelsVideo, 3D & Audio Generative ModelsMedium
⚡Flashcard Drill➔Mastery:
Neural Radiance Fields (NeRF) vs 3D Gaussian Splatting: Continuous Volumetric vs Discrete Primitives
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
NeRF models 3D scenes as continuous implicit neural volumetric functions optimized via ray marching, while 3D Gaussian Splatting represents scenes using explicit anisotropic 3D Gaussians rendered via fast tile-based rasterization.
📌 Key Takeaways
- •NeRF representation (Mildenhall et al.): encodes continuous scene geometry and view-dependent color using a coordinate MLP , rendering via numerical volume integration along camera rays
- •3D Gaussian Splatting (Kerbl et al.): models scenes using millions of explicit anisotropic 3D Gaussians defined by position, covariance matrix, opacity, and spherical harmonics
- •Rendering speed breakthrough: NeRF requires evaluating hundreds of MLP forward passes per pixel (seconds per frame), while 3DGS achieves real-time FPS rendering via GPU tile-based rasterization
📐 Mathematical Derivations
Mathematical Mechanism: 1. NeRF Volumetric Ray Marching: For camera ray , rendered pixel color integrates continuous volume density and color between near bound and far bound : Approximated numerically via stratified quadrature sampling across discrete points: 2. 3D Gaussian Splatting Formulation: A 3D Gaussian is parameterized by mean position and 3D covariance matrix : To ensure positive semi-definiteness during gradient descent, covariance is decomposed into scaling vector and rotation quaternion : 3. 2D Projective Splatting (Zwicker et al.): Given viewing transformation and projective Jacobian , the 2D projected screen covariance is: Pixels are colored via tile-based alpha-blending of sorted Gaussians:
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Paradigm Shift from Implicit to Explicit: NeRF was revolutionary in 2020 for eliminating discrete polygonal meshes and achieving photorealistic novel view synthesis. However, querying a deep MLP 128 times per ray for a frame ( rays) requires hundreds of millions of neural network evaluations per image, capping rendering speed at 0.1-1 FPS. 3D Gaussian Splatting eliminates neural networks at inference entirely: rendering is pure GPU rasterization (sorting 2D Gaussians and executing hardware alpha blending), reaching 100-200 FPS on consumer hardware at equal or higher visual quality. ② Memory Footprint and Storage: (a) NeRF: Compact representation; the entire scene is compressed into a MLP weight checkpoint. (b) 3DGS: Storing 2-5 million 3D Gaussians (each storing position, rotation, scale, opacity, and 16 spherical harmonic coefficients) consumes to of RAM/disk storage. Compression techniques (vector quantization of spherical harmonics, Gaussian pruning) reduce footprint by . ③ Adaptive Density Control in 3DGS: 3DGS periodically splits large Gaussians in under-reconstructed regions and clones small Gaussians in high-frequency detail regions, while pruning transparent Gaussians (). ⑤ Interview Strategy: Contrast implicit volumetric MLP rendering against explicit Gaussian primitive rasterization, write NeRF's volume rendering quadrature equation, derive 3DGS covariance projection , and compare rendering speed vs memory footprint.
⚠️ Common Interview Pitfalls
- ✕Attempting real-time interactive rendering with standard vanilla NeRF; vanilla NeRF requires seconds to render a single frame
- ✕Optimizing covariance matrix directly without scale-rotation decomposition (), causing non-positive-definite covariance collapse
- ✕Underestimating the storage footprint of raw uncompressed 3D Gaussian splat files () in web deployment pipelines
🎯 Interviewer Follow-up Inquiries
- ?Why is the scale-rotation decomposition mathematically necessary when optimizing 3D Gaussian Splatting?
- ?How does tile-based GPU rasterization enable 3D Gaussian Splatting to render complex scenes at over 100 FPS?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚