M9-018M9: Senior ML & AI System DesignFeature Store, Sharding & CI InfraMedium
⚡Flashcard Drill➔Mastery:

Design a Layout-Aware Document ETL & Parent-Child RAG Indexing Pipeline

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M9-019 →
📐 Mathematical Definition
Small Child(200 tok)→Dense RetrieveFull Parent(1500 tok)+Breadcrumb Prefix\text{Small Child(200 tok)} \xrightarrow{\text{Dense Retrieve}} \text{Full Parent(1500 tok)} + \text{Breadcrumb Prefix}
⚡ One-Sentence Core Answer
Cure chunk-boundary collapse with layout-aware parsing (Docling), header breadcrumb prefixes, sharp 200-token child index vectors, and rich 1,500-token parent expansions.

📌 Key Takeaways

  • •
    L1 Boundary Collapse: naive character splitting divorces table headers from values and scrambles columns
  • •
    L2 Table Indexing: index concise natural-language table summaries linked to intact Markdown payloads
  • •
    L2 Small-to-Big Index: score crisp 200-token child chunks cleanly, hydrate full 1,500-token parent sections
  • •
    L3 Hierarchy Prefixes: inject H1/H2 title breadcrumbs into every child so pronouns never lose subject context

📐 Mathematical Derivations

Separating index granularity from synthesis granularity optimizes both retrieval SNR and generation coherence simultaneously.

🏭 Production Trade-offs

Persist page-level Docling Markdown into a Silver lakehouse layer so chunking/embedding iterations avoid repeating slow OCR.
⚠️ Common Interview Pitfalls
  • ✕
    Extracting two-column PDFs with layout-blind line readers
  • ✕
    Assuming minor token overlap fixes multi-page table header loss
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you propagate header rows automatically when splitting an 8-page spreadsheet?
  • ?
    How do you measure chunking gains directly via retrieval chunk-boundary recall?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM9-017: Design a 10-Billion-Scale Distributed Vector Search Architecture📋 Back to BankNext →M9-019: Design a Shadow-Traffic & Canary Auto-Rollback Release Gate for LLMs