M8-031M8: ML Systems, Engineering & ResearchInference Serving & DeploymentEasy
⚡Flashcard Drill➔Mastery:

Explain Common Model Serving Deployment Paradigms and Architectural Patterns

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M8-032 →
📐 Mathematical Definition
forms: online, batch, streaming, edge;choose by latency/throughput\text{forms}:\ \text{online},\ \text{batch},\ \text{streaming},\ \text{edge};\qquad \text{choose by latency/throughput}
⚡ One-Sentence Core Answer
Model serving spans four primary paradigms—Online synchronous, Offline batch, Streaming event-driven, and Edge on-device—often combined into multi-stage cascades (such as offline candidate pre-computation paired with online real-time re-ranking) to balance latency, throughput, cost, and privacy.

📌 Key Takeaways

  • •
    Four core deployment paradigms: Online/Real-time (synchronous, sub-second latency SLAs), Offline Batch (asynchronous high-throughput scoring), Streaming/Nearline (continuous event-driven inference), and Edge/On-device (local low-latency and privacy-preserving).
  • •
    Hybrid & cascade topologies: Offline batch pre-generation of recommendation candidates combined with online real-time fine-ranking; edge coarse filtering coupled with cloud deep inference.
  • •
    Selection criteria: Latency tolerance (milliseconds vs. hours), data security/privacy boundaries, throughput scale, compute resource constraints, and deployment cost.

📐 Mathematical Derivations

Taxonomy of Deployment Paradigms:

(1) Online Synchronous Serving:
- Execution Model: Client issues synchronous RPC (gRPC / HTTP REST) and blocks waiting for immediate response.
- SLA Constraints: Strict latency budgets (e.g., 10−50 ms10 - 50\text{ ms} for recommendation ranking; <2 s< 2\text{ s} TTFT for interactive LLMs); 99.9%+ availability.
- Infrastructure: Model serving frameworks (NVIDIA Triton, vLLM, TorchServe) hosted on auto-scaling GPU clusters with load balancers and horizontal pod autoscalers (HPA).

(2) Offline Batch Serving:
- Execution Model: Scheduled jobs (Airflow DAGs, Spark jobs) processing millions of records asynchronously from distributed object stores.
- Characteristics: Latency-insensitive (hours/days turnaround); maximizes vectorized GPU throughput via massive batch sizes; compute instances spin up on demand and terminate upon completion.
- Use Cases: Daily user embedding generation, batch churn scoring, catalog text classification, offline dataset labeling.

(3) Streaming / Nearline Serving:
- Execution Model: Event-driven processors (Apache Flink, Kafka Streams) consuming continuous message streams, performing micro-batched or per-event inference.
- Characteristics: Medium latency budget (100 ms−5 s100\text{ ms} - 5\text{ s}); decouple user-facing HTTP request paths from heavy model scoring.
- Use Cases: Real-time fraud detection, financial transaction scoring, live content moderation, IoT sensor telemetry.

(4) Edge / On-Device Serving:
- Execution Model: Executing inference locally on client hardware (smartphones, IoT microcontrollers, automotive ECUs).
- Benefits: Zero network latency, zero cloud hosting server costs, offline availability, and absolute privacy (user data never leaves the device).
- Hardware Constraints: Stringent thermal, power, and memory limits; requires aggressive quantization (INT4/INT8 via CoreML, ONNX Runtime, llama.cpp, TensorRT-Edge).

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① The industrial hybrid pattern: Offline pre-computation + Online fine-ranking—calculating full deep recommendations across 100 million items in real-time is computationally impossible within 30ms; systems pre-compute candidate vectors in daily batch jobs, store them in Redis/ANN indexes, and invoke deep neural networks online only for the top 500 candidates. ② Batch serving is exponentially cheaper—if business requirements can tolerate a 4-hour delay, deploying an online service wastes massive continuous cluster idle costs; batch jobs running on spot instances reduce serving costs by 80%. ③ Edge serving operational hurdles—edge models cannot be instantly rolled back or hot-swapped (requiring app store releases or OTA firmware pushes); fragmented client hardware platforms demand extensive cross-compilation testing. ④ Model cascades (Small model filter -> Big model compute)—routing 100% of user queries to a 70B parameter LLM is cost-prohibitive; systems deploy a lightweight 1B model on CPU/edge to filter out 70% of benign or simple queries, routing only complex requests to cloud GPU clusters. ⑤ Streaming nearline as a protective buffer—placing heavy risk/fraud inference on asynchronous Kafka topics protects front-end checkout services from crashing when model clusters experience temporary latency spikes. ⑥ Interview takeaway—systematically compare the four paradigms across latency, cost, and complexity, detail the hybrid 'offline pre-compute + online rank' pattern, and articulate model cascade filtering.
⚠️ Common Interview Pitfalls
  • ✕
    Deploying high-cost online synchronous serving infrastructure for analytical workloads that users inspect only once daily.
  • ✕
    Ignoring thermal, battery, and memory constraints when porting server-side deep neural networks to client edge devices.
  • ✕
    Tightly coupling synchronous microservices to heavy deep learning inference without timeout fallbacks, causing cascading user-facing 504 gateway timeouts.
🎯 Interviewer Follow-up Inquiries
  • ?
    How do enterprise recommendation systems partition work between offline batch candidate generation and online real-time re-ranking?
  • ?
    What quantization and operator compilation techniques enable running 7B parameter LLMs locally on mobile hardware?
  • ?
    How does an asynchronous streaming nearline scoring topology protect synchronous checkout APIs from latency spikes?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM8-030: Explain the Core Metrics, Latency Decomposition, and Goodput for Online Inference Serving📋 Back to BankNext →M8-032: Explain the Architecture, Roles, and Synergies of Inference Engines: Triton vs. TensorRT vs. vLLM