Stage 2 · Incremental CDC Document Sync, Content-Hash Dedup & Tombstone GC Pattern
Production ingestion pattern replacing brittle nightly full re-indexes with event-driven CDC, content-hash diffing, blue/green alias swaps on schema shifts, and atomic tombstone deletion for revoked or edited documents.
Best For
High-churn enterprise repositories (Confluence, Drive, S3, CRM tickets) where stale deleted snippets violate compliance and nightly full re-embedding is cost-prohibitive.
Human-AI Boundary
Embedding model migrations trigger background dual-write shadow collections verified offline before engineers cut over collection aliases atomically.
Anti-Patterns to Avoid
- Re-embedding the entire multi-million-chunk corpus every night when <2% of documents changed, wasting 50x GPU/API spend
- Appending newly edited document versions into the vector store without deleting prior chunk_ids for that doc_id, poisoning answers with outdated policies
Latency/Cost Profile & Eval Metrics
Incremental CDC sync SLA < 60s from source commit/delete to index visibility · Hash comparison SQLite/Postgres metadata table vs skipped unchanged chunk embeddings
- Deletion & ACL Propagation SLA: < 60s P99 from upstream document deletion/revocation to zero searchability (Run synthetic delete-and-query probe every 5 minutes)
- Redundant Re-Embedding Ratio: < 1% unchanged chunks re-embedded during scheduled reconciliation (Track embed API calls against actual source diff count)
💰 Business ROI Model & Evidence Level
production_practiceOn a 2M-chunk corpus with 1.5% daily churn, content-hash CDC skips 98.5% of redundant parsing and embedding calls—cutting daily refresh spend from ~$400 to <$8 while preventing stale-policy hallucinations.
🏗️ End-to-End Architecture Pipeline
Consume source webhooks/outbox cursors and short-circuit unchanged doc_sha256 + acl_version pairs.
Insert new chunk hashes first for zero-downtime reads, then atomically delete obsolete_chunk_ids and commit manifest version.
⚖️ Engineering Trade-Off & Decision Matrix
🚨 Production Failure Modes & Signatures
- Out-of-order delete followed by delayed upsert resurrecting deleted chunks without monotonic source_version checks
- High HNSW fragmentation after 30%+ in-place deletes without periodic segment compaction
🎤 Interview Defense & System Design Deep-Dive Q&A
- How do you guarantee a document deleted in SharePoint cannot be retrieved or leaked even if an incremental sync worker crashes mid-update?
Bidirectional Taxonomy Links
Internal Tools & Primary Sources
- Debezium Outbox & Change Data Capture Architecture (official_docs, 2026-11-17)
- Qdrant Point Deletion & Collection Aliasing Docs (official_docs, 2026-11-17)