← Back to Registry Hub/Site Search/L1 · stage2_data_ingestion_pattern
⚡Flashcard Drill➔
L1S2 · Data & RetrievalS6 · Security & FinOpsVerified: 2026-11-17

Stage 2 · Incremental CDC Document Sync, Content-Hash Dedup & Tombstone GC Pattern

Production ingestion pattern replacing brittle nightly full re-indexes with event-driven CDC, content-hash diffing, blue/green alias swaps on schema shifts, and atomic tombstone deletion for revoked or edited documents.

Best For

High-churn enterprise repositories (Confluence, Drive, S3, CRM tickets) where stale deleted snippets violate compliance and nightly full re-embedding is cost-prohibitive.

Human-AI Boundary

Embedding model migrations trigger background dual-write shadow collections verified offline before engineers cut over collection aliases atomically.

Anti-Patterns to Avoid

  • Re-embedding the entire multi-million-chunk corpus every night when <2% of documents changed, wasting 50x GPU/API spend
  • Appending newly edited document versions into the vector store without deleting prior chunk_ids for that doc_id, poisoning answers with outdated policies

Latency/Cost Profile & Eval Metrics

Incremental CDC sync SLA < 60s from source commit/delete to index visibility · Hash comparison SQLite/Postgres metadata table vs skipped unchanged chunk embeddings

  • Deletion & ACL Propagation SLA: < 60s P99 from upstream document deletion/revocation to zero searchability (Run synthetic delete-and-query probe every 5 minutes)
  • Redundant Re-Embedding Ratio: < 1% unchanged chunks re-embedded during scheduled reconciliation (Track embed API calls against actual source diff count)

💰 Business ROI Model & Evidence Level

production_practice

On a 2M-chunk corpus with 1.5% daily churn, content-hash CDC skips 98.5% of redundant parsing and embedding calls—cutting daily refresh spend from ~$400 to <$8 while preventing stale-policy hallucinations.

🏗️ End-to-End Architecture Pipeline

Step 1: CDC Event Capture & Document Manifest Diff
≤ 500msZero LLM cost

Consume source webhooks/outbox cursors and short-circuit unchanged doc_sha256 + acl_version pairs.

Step 2: Chunk-Level Set Diff, Idempotent Upsert & Tombstone GC
≤ 4000msOnly delta chunks embedded

Insert new chunk hashes first for zero-downtime reads, then atomically delete obsolete_chunk_ids and commit manifest version.

⚖️ Engineering Trade-Off & Decision Matrix

Incremental CDC + Content-Hash Manifest
When to Use: Live enterprise knowledge bases with hourly edits, deletes, or ACL updates.
Trade-Off: Requires a lightweight state manifest table and monotonic version stamps to guard against out-of-order CDC events.
Blue/Green Full Shadow Collection Swap
When to Use: Upgrading embedding models, vector dimensions, or chunking schemas.
Trade-Off: Requires temporary 2x storage during backfill in exchange for atomic alias rollback.

🚨 Production Failure Modes & Signatures

  • Out-of-order delete followed by delayed upsert resurrecting deleted chunks without monotonic source_version checks
  • High HNSW fragmentation after 30%+ in-place deletes without periodic segment compaction

🎤 Interview Defense & System Design Deep-Dive Q&A

  1. How do you guarantee a document deleted in SharePoint cannot be retrieved or leaked even if an incremental sync worker crashes mid-update?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

Internal Tools & Primary Sources