M8-019M8: ML Systems, Engineering & ResearchData Pipelines & StreamingEasy
⚡Flashcard Drill➔Mastery:
Explain the Architectural Differences and Selection Criteria Between Batch and Stream Processing
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Batch processing optimizes high-throughput computation over bounded historical data with trivial re-runnability, whereas stream processing minimizes end-to-end latency over unbounded continuous event streams at the cost of stateful complexity, checkpointing, and watermark handling.
📌 Key Takeaways
- •Batch processing: Scheduled execution over bounded datasets, maximizing compute throughput and resource efficiency with trivial idempotency and reprocessing.
- •Stream processing: Event-driven continuous execution over unbounded streams, delivering sub-second latency via complex state backends (RocksDB) and watermark management.
- •Architecture selection: Governed by business latency budgets (seconds vs. hours), operational complexity tolerances, state storage footprints, and cost structures.
📐 Mathematical Derivations
Theoretical & Architectural Comparison:
(1) Batch Processing Engine Paradigm:
- Data Semantics: Operates on bounded, immutable datasets delineated by discrete time boundaries.
- Resource Model: Ephemeral compute allocation (e.g., Spark on YARN/Kubernetes) scaled dynamically to maximize vectorized bulk I/O throughput.
- Failure Recovery: Lineage graph recomputation (RDD lineage); trivial re-runnability by reprocessing the bounded partition.
(2) Stream Processing Engine Paradigm:
- Data Semantics: Ingests unbounded continuous sequences where event time may diverge significantly from ingestion processing time .
- State Management: Continuous state maintenance (e.g., key-value states in embedded RocksDB) persisting session aggregations across millions of concurrent keys.
- Event-Time Watermarking: Monotonically increasing timestamps establishing an arrival guarantee: with probability , bounding late-data handling.
- Fault Tolerance: Asynchronous distributed barrier snapshotting (Chandy-Lamport algorithm / Flink checkpoints) coupled with two-phase commit sinks to enforce end-to-end exactly-once semantics.
(3) Architectural Evolution & Unification:
- Lambda Architecture: Parallel batch (Hadoop/Spark for accuracy) and speed (Storm/Flink for real-time) layers unified at the serving layer; burdened by dual codebase maintenance and semantic drift.
- Kappa Architecture: Solely event-stream driven; reprocessing historical data by replaying Kafka/Pulsar log offsets through newly spawned stream processor instances.
- Unified Storage & Compute: Unified stream-batch engines (Apache Flink, Spark Structured Streaming) exposing identical relational APIs over bounded tables and unbounded streams.
(1) Batch Processing Engine Paradigm:
- Data Semantics: Operates on bounded, immutable datasets delineated by discrete time boundaries.
- Resource Model: Ephemeral compute allocation (e.g., Spark on YARN/Kubernetes) scaled dynamically to maximize vectorized bulk I/O throughput.
- Failure Recovery: Lineage graph recomputation (RDD lineage); trivial re-runnability by reprocessing the bounded partition.
(2) Stream Processing Engine Paradigm:
- Data Semantics: Ingests unbounded continuous sequences where event time may diverge significantly from ingestion processing time .
- State Management: Continuous state maintenance (e.g., key-value states in embedded RocksDB) persisting session aggregations across millions of concurrent keys.
- Event-Time Watermarking: Monotonically increasing timestamps establishing an arrival guarantee: with probability , bounding late-data handling.
- Fault Tolerance: Asynchronous distributed barrier snapshotting (Chandy-Lamport algorithm / Flink checkpoints) coupled with two-phase commit sinks to enforce end-to-end exactly-once semantics.
(3) Architectural Evolution & Unification:
- Lambda Architecture: Parallel batch (Hadoop/Spark for accuracy) and speed (Storm/Flink for real-time) layers unified at the serving layer; burdened by dual codebase maintenance and semantic drift.
- Kappa Architecture: Solely event-stream driven; reprocessing historical data by replaying Kafka/Pulsar log offsets through newly spawned stream processor instances.
- Unified Storage & Compute: Unified stream-batch engines (Apache Flink, Spark Structured Streaming) exposing identical relational APIs over bounded tables and unbounded streams.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① Latency requirement dictates architecture—many enterprise applications genuinely require only 5-minute freshness; forcing sub-second stream processing introduces immense operational overhead (watermarks, checkpoint failures, backpressure tuning) without business justification. ② State backend scalability limits—in stream processing, holding 30-day user rolling window state in RocksDB exhausts memory/disk and complicates state migration; batch compute effortlessly calculates multi-month aggregations in bulk. ③ Backpressure and burst absorption—streaming systems must handle downstream database saturation through credit-based flow control or dynamic Kafka partition buffering, whereas batch pipelines absorb load via task queue scheduling. ④ Lambda codebase duplication vs. Kappa replay costs—Lambda introduces permanent synchronization bugs across Java/Scala streaming and SQL batch code; Kappa eliminates code duplication but incurs substantial I/O costs when replaying 6 months of historical logs from cold tiers. ⑤ Micro-batching as a pragmatic sweet spot—engines like Spark Structured Streaming running at 500ms intervals deliver 90% of the operational simplicity of batch processing with acceptable low-latency responsiveness. ⑥ Interview takeaway—contrast bounded vs. unbounded data paradigms, articulate the Chandy-Lamport checkpointing and watermark mechanics for streaming, and emphasize that latency requirements and operational simplicity must govern the selection.
⚠️ Common Interview Pitfalls
- ✕Over-engineering a complex sub-second streaming pipeline when business metrics and user-facing dashboards refresh only on an hourly basis.
- ✕Failing to account for out-of-order events and late data arrival in streaming pipelines, leading to silent state drops or corrupted aggregations.
- ✕Neglecting backpressure propagation in streaming pipelines, causing out-of-memory crashes during upstream traffic surges.
🎯 Interviewer Follow-up Inquiries
- ?How does Apache Flink's asynchronous barrier snapshotting (Chandy-Lamport) achieve zero-downtime checkpoints?
- ?Under what data volume conditions does Kappa log replay become economically or operationally unfeasible compared to batch re-computation?
- ?How do watermark delay thresholds mathematically balance computation latency against late-data drop rates?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚