4.2
4.2 Multimodal, Speech & Generative Media Models
Vision-Language (CLIP/LLaVA/Qwen-VL), cross-attention multimodal projectors, Diffusion Transformers (DiT) for video, and native full-duplex speech models.
Vision-Language Models (VLM) & Cross-Modal Alignment
ViT visual encoders (SigLIP/CLIP), cross-attention multimodal projection layers, visual instruction tuning (LLaVA/Qwen-VL), dynamic high-res image patch splitting, and MMBench evaluation.
🏢 Companies
OpenAI (GPT-4o)、Google DeepMind (Gemini)、Meta AI、Qwen (Qwen-VL)
🛠️ Tech Stack
VLMLLaVACLIPSigLIPMultimodal ProjectorQwen-VLDynamic Patching
💼 Roles & Salary
Multimodal Engineer、Computer Vision Engineer、Research Scientist
💰 $240K - $520K / year (VLM & Multimodal) | ¥700K - ¥1.9M / year
📚 Prerequisites: Vision Transformer (ViT) & Patch Embedding • Contrastive Learning Loss Derivations • Multimodal Projection & Alignment Layers • Multimodal Benchmark Suites (MMBench)
Diffusion / DiT Video Generation & Native Speech AI
Diffusion Transformers (DiT), Flow Matching, 3D spatiotemporal VAE compression, native end-to-end full-duplex speech-to-speech models, and generated media provenance watermarking.
🏢 Companies
Midjourney、Runway、Kling (快手)、Pika、ElevenLabs、Suno
🛠️ Tech Stack
DiTDiffusionFlow MatchingVideo GenerationAudio CodecNative Speech LLMMidjourney
💼 Roles & Salary
Generative Media Researcher、Computer Vision Engineer、Speech Engineer
💰 $250K - $540K / year (Generative Media & Speech) | ¥750K - ¥2.0M / year
📚 Prerequisites: Diffusion Math (DDPM/DDIM/Score-based) • Flow Matching & Continuous ODE Solvers • 3D Spatiotemporal VAE Downsampling • Neural Audio Codecs & Full-Duplex Speech