M8-019M8: ML Systems, Engineering & ResearchData Pipelines & StreamingEasy
⚡Flashcard Drill➔Mastery:

Explain the Architectural Differences and Selection Criteria Between Batch and Stream Processing

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M8-020 →
📐 Mathematical Definition
batch: window=hours/days;stream: window=seconds\text{batch}:\ \text{window}=\text{hours/days};\qquad \text{stream}:\ \text{window}=\text{seconds}
⚡ One-Sentence Core Answer
Batch processing optimizes high-throughput computation over bounded historical data with trivial re-runnability, whereas stream processing minimizes end-to-end latency over unbounded continuous event streams at the cost of stateful complexity, checkpointing, and watermark handling.

📌 Key Takeaways

  • •
    Batch processing: Scheduled execution over bounded datasets, maximizing compute throughput and resource efficiency with trivial idempotency and reprocessing.
  • •
    Stream processing: Event-driven continuous execution over unbounded streams, delivering sub-second latency via complex state backends (RocksDB) and watermark management.
  • •
    Architecture selection: Governed by business latency budgets (seconds vs. hours), operational complexity tolerances, state storage footprints, and cost structures.

📐 Mathematical Derivations

Theoretical & Architectural Comparison:

(1) Batch Processing Engine Paradigm:
- Data Semantics: Operates on bounded, immutable datasets D={x1,…,xN}\mathcal{D} = \{x_1, \dots, x_N\} delineated by discrete time boundaries.
- Resource Model: Ephemeral compute allocation (e.g., Spark on YARN/Kubernetes) scaled dynamically to maximize vectorized bulk I/O throughput.
- Failure Recovery: Lineage graph recomputation (RDD lineage); trivial re-runnability by reprocessing the bounded partition.

(2) Stream Processing Engine Paradigm:
- Data Semantics: Ingests unbounded continuous sequences S=⟨(e1,t1),(e2,t2),… ⟩S = \langle (e_1, t_1), (e_2, t_2), \dots \rangle where event time teventt_{\text{event}} may diverge significantly from ingestion processing time tproct_{\text{proc}}.
- State Management: Continuous state maintenance (e.g., key-value states in embedded RocksDB) persisting session aggregations across millions of concurrent keys.
- Event-Time Watermarking: Monotonically increasing timestamps W(t)W(t) establishing an arrival guarantee: ∀e∈Stream,tevent(e)≤W(t)\forall e \in \text{Stream}, t_{\text{event}}(e) \le W(t) with probability 1−ϵ1 - \epsilon, bounding late-data handling.
- Fault Tolerance: Asynchronous distributed barrier snapshotting (Chandy-Lamport algorithm / Flink checkpoints) coupled with two-phase commit sinks to enforce end-to-end exactly-once semantics.

(3) Architectural Evolution & Unification:
- Lambda Architecture: Parallel batch (Hadoop/Spark for accuracy) and speed (Storm/Flink for real-time) layers unified at the serving layer; burdened by dual codebase maintenance and semantic drift.
- Kappa Architecture: Solely event-stream driven; reprocessing historical data by replaying Kafka/Pulsar log offsets through newly spawned stream processor instances.
- Unified Storage & Compute: Unified stream-batch engines (Apache Flink, Spark Structured Streaming) exposing identical relational APIs over bounded tables and unbounded streams.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① Latency requirement dictates architecture—many enterprise applications genuinely require only 5-minute freshness; forcing sub-second stream processing introduces immense operational overhead (watermarks, checkpoint failures, backpressure tuning) without business justification. ② State backend scalability limits—in stream processing, holding 30-day user rolling window state in RocksDB exhausts memory/disk and complicates state migration; batch compute effortlessly calculates multi-month aggregations in bulk. ③ Backpressure and burst absorption—streaming systems must handle downstream database saturation through credit-based flow control or dynamic Kafka partition buffering, whereas batch pipelines absorb load via task queue scheduling. ④ Lambda codebase duplication vs. Kappa replay costs—Lambda introduces permanent synchronization bugs across Java/Scala streaming and SQL batch code; Kappa eliminates code duplication but incurs substantial I/O costs when replaying 6 months of historical logs from cold tiers. ⑤ Micro-batching as a pragmatic sweet spot—engines like Spark Structured Streaming running at 500ms intervals deliver 90% of the operational simplicity of batch processing with acceptable low-latency responsiveness. ⑥ Interview takeaway—contrast bounded vs. unbounded data paradigms, articulate the Chandy-Lamport checkpointing and watermark mechanics for streaming, and emphasize that latency requirements and operational simplicity must govern the selection.
⚠️ Common Interview Pitfalls
  • ✕
    Over-engineering a complex sub-second streaming pipeline when business metrics and user-facing dashboards refresh only on an hourly basis.
  • ✕
    Failing to account for out-of-order events and late data arrival in streaming pipelines, leading to silent state drops or corrupted aggregations.
  • ✕
    Neglecting backpressure propagation in streaming pipelines, causing out-of-memory crashes during upstream traffic surges.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Apache Flink's asynchronous barrier snapshotting (Chandy-Lamport) achieve zero-downtime checkpoints?
  • ?
    Under what data volume conditions does Kappa log replay become economically or operationally unfeasible compared to batch re-computation?
  • ?
    How do watermark delay thresholds mathematically balance computation latency against late-data drop rates?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM8-018: Describe the Architecture and Engineering Principles of a Typical Offline Data Pipeline📋 Back to BankNext →M8-020: Explain the Dimensions and Technical Implementation of Data Quality Monitoring