Data Pipelines & Streaming

Data pipelines and streaming. Kafka acts as the message bus (ordered per partition, append-only writes + page cache + zero-copy for high throughput; at-least-once / exactly-once consumption semantics), and Flink as a stateful stream engine (event-time windows, watermarks, state backends and checkpoints for exactly-once), emitting second-level real-time features for online inference and feeding a Feature Store aligned with offline features. Batch-stream unification: Lambda (separate batch and stream pipelines — double maintenance and consistency drift) versus Kappa (a single stream pipeline, with batch treated as historical replay of the stream). Data platform layering: ODS (raw ingestion) → DWD (cleaning, normalization, redaction) → DWS (wide tables and aggregated metrics for serving), with lakehouse formats (Iceberg/Hudi/Delta) unifying batch and stream storage. The watermark $W = \max_{\text{seen}}(t_{\text{event}}) - \varepsilon$ governs out-of-order and late data, where $\varepsilon$ is the maximum allowed disorder latency.

online ML systems (recommendation, risk control, ads) need second-level features; platform layering and warehouse modeling are classic system-design questions; interviews ask about Kafka's throughput mechanics and message semantics, Flink exactly-once, Lambda vs Kappa, and the duties of ODS/DWD/DWS.

versus daily offline batch, end-to-end latency drops from days to seconds (the real-time chain finishes in about one second), letting online models consume the latest user behavior; Kappa unifies batch and stream via replay, removing Lambda's inconsistency and double maintenance; layering unifies metric definitions, clarifies lineage and keeps compliance controllable (redaction at DWD, metric unification at DWS).

Test Your Knowledge: Practice Questions for "Data Pipelines & Streaming"

Single choice pitfall questions with instant feedback and mistake tracking.