04. Foundation Models & Model Assets
OpenAI
Anthropic
Google DeepMind
Meta (Llama)
Qwen 通义千问
快手 (可灵 Kling)
Hugging Face
魔搭 (ModelScope)OpenAIGoogle DeepMindMeta AIQwen 通义千问Developing core frontier assets: Dense/MoE Transformer architectures, Long-Context reasoning, Multimodal generative media, and open model checkpoints.
4.1 Foundation LLM & Reasoning Architectures
Transformer architecture evolutions, RoPE long-context scaling, Mixture-of-Experts (MoE) sparse routing, DeepSeek MLA attention, and chain-of-thought reasoning models.
Transformer Evolution & Long-Context Scaling
Dense Transformer foundations, KV cache architectures (MHA/GQA/MQA), RoPE position embeddings with YaRN/ALiBi long-context extrapolation, RMSNorm, SwiGLU, and BPE tokenizer designs.
OpenAIAnthropicGoogle DeepMindQwen 通义千问Meta (Llama)MoE Sparse Routing & MLA Reasoning Architecture
Top-k sparse expert routing, auxiliary-loss-free dynamic load balancing, DeepSeek Multi-Head Latent Attention (MLA) low-rank KV compression, and chain-of-thought reasoning models.
Meta (Llama)Google DeepMindOpenAIQwen 通义千问4.2 Multimodal, Speech & Generative Media Models
Vision-Language (CLIP/LLaVA/Qwen-VL), cross-attention multimodal projectors, Diffusion Transformers (DiT) for video, and native full-duplex speech models.
Vision-Language Models (VLM) & Cross-Modal Alignment
ViT visual encoders (SigLIP/CLIP), cross-attention multimodal projection layers, visual instruction tuning (LLaVA/Qwen-VL), dynamic high-res image patch splitting, and MMBench evaluation.
OpenAIGoogle DeepMindMeta AIQwen 通义千问Diffusion / DiT Video Generation & Native Speech AI
Diffusion Transformers (DiT), Flow Matching, 3D spatiotemporal VAE compression, native end-to-end full-duplex speech-to-speech models, and generated media provenance watermarking.
快手 (可灵 Kling)4.3 Open Model Assets, Licenses & Checkpoint Ecosystem
Open model registries (Hugging Face / ModelScope), SafeTensors serialization, LoRA/QLoRA adapter assets, Model Card metadata, and open source license governance.
Open Model Hubs, SafeTensors & Adapter Ecosystem
Open weights distribution on Hugging Face & ModelScope, SafeTensors zero-copy secure mmap serialization, LoRA/QLoRA adapter asset management, and universal embedding/reranker artifacts.
Hugging Face魔搭 (ModelScope)Meta (Llama)Qwen 通义千问Model Card Specs, License Governance & Security Scanning
Model Card metadata standards, license governance (Apache 2.0 / MIT / Llama Community License terms), SHA256 integrity checksums, and pickle malware security scanning.
Hugging FaceMeta (Llama)