M3-101M3: Deep Learning FoundationsConvolution & Vision FoundationsHard
⚡Flashcard Drill➔Mastery:
Architectural Design Philosophies: ViT vs Swin Transformer vs ConvNeXt
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
ViT applies minimal-bias global attention (data-hungry); Swin reintroduces hierarchical shifted windows; ConvNeXt modernizes pure CNNs with Transformer design principles.
📌 Key Takeaways
- •ViT (Dosovitskiy et al.): non-hierarchical, global self-attention across patches; requires massive pretraining datasets (JFT-300M)
- •Swin (Liu et al.): hierarchical multi-scale feature maps, linear complexity via shifted local windows; general-purpose dense vision backbone
- •ConvNeXt (Liu et al.): purely convolutional ( depthwise convs, inverted bottleneck, LayerNorm); matches Swin accuracy with superior simplicity
📐 Mathematical Derivations
Philosophical and Structural Comparison:
① Vision Transformer (ViT, ICLR 2021):
- Philosophy: Strip away all 2D vision inductive biases. Treat images as sequences of 1D word tokens ( patches).
- Complexity: global self-attention where . Scale-invariant columnar architecture (single resolution).
- Limitation: Quadratic complexity in makes high-resolution dense prediction (detection/segmentation) computationally prohibitive.
② Swin Transformer (ICCV 2021 Best Paper):
- Philosophy: Reintroduce hierarchy and spatial locality into attention.
- Mechanism: Partitions patches into local windows (). Computes self-attention within windows ( linear complexity). Alternates with Shifted Windows (SW-MSA) to enable cross-window information exchange. Generates multi-scale feature pyramids ( downsampling).
③ ConvNeXt (CVPR 2022):
- Philosophy: 'Can modern CNNs compete with Transformers without attention?'
- Recipe: Modernizes ResNet step-by-step: (1) Patchify stem ( stride 4); (2) Inverted bottleneck ( channels); (3) Large depthwise kernels (matching attention windows); (4) Micro-design: GELU, fewer activations, LayerNorm instead of BatchNorm, separate downsampling layers. Proves attention is not strictly mandatory for competitive representation learning.
① Vision Transformer (ViT, ICLR 2021):
- Philosophy: Strip away all 2D vision inductive biases. Treat images as sequences of 1D word tokens ( patches).
- Complexity: global self-attention where . Scale-invariant columnar architecture (single resolution).
- Limitation: Quadratic complexity in makes high-resolution dense prediction (detection/segmentation) computationally prohibitive.
② Swin Transformer (ICCV 2021 Best Paper):
- Philosophy: Reintroduce hierarchy and spatial locality into attention.
- Mechanism: Partitions patches into local windows (). Computes self-attention within windows ( linear complexity). Alternates with Shifted Windows (SW-MSA) to enable cross-window information exchange. Generates multi-scale feature pyramids ( downsampling).
③ ConvNeXt (CVPR 2022):
- Philosophy: 'Can modern CNNs compete with Transformers without attention?'
- Recipe: Modernizes ResNet step-by-step: (1) Patchify stem ( stride 4); (2) Inverted bottleneck ( channels); (3) Large depthwise kernels (matching attention windows); (4) Micro-design: GELU, fewer activations, LayerNorm instead of BatchNorm, separate downsampling layers. Proves attention is not strictly mandatory for competitive representation learning.
🏭 Production Trade-offs
Deployment choices: ViT remains standard in multi-modal foundational models (CLIP, LLaVA) where global cross-attention with language is primary. Swin is favored for hierarchical detection/segmentation. ConvNeXt is favored for high-throughput edge deployment where standard Conv2D kernels have mature TensorRT/mobile optimization.
⚠️ Common Interview Pitfalls
- ✕Deploying standard ViT on detection images, triggering memory explosion in attention
- ✕Assuming ConvNeXt requires attention mechanisms; ConvNeXt uses 100% standard convolutional operations
🎯 Interviewer Follow-up Inquiries
- ?How does Swin Transformer's shifted window masking mechanism enable cross-window self-attention without increasing the number of windows?
- ?What specific micro-architectural modifications in ConvNeXt bridge the performance gap with Swin Transformer?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚