← Back to Registry Hub/Site Search/L3 · reference_project
⚡Flashcard Drill➔
L3S2 · Data & RetrievalS3 · Model & AgentS4 · UX & HITLS5 · Eval & LLMOpsS6 · Security & FinOpsVerified: 2026-11-15

Verifiable-Citation SEC 10-K Research Copilot (Reference Portfolio Project)

High-signal portfolio reference implementation combining Docling filing ingestion, BM25 + dense RRF + BGE-Reranker retrieval, LangGraph deterministic math verification, and a 60-query numerical faithfulness CI eval suite.

Best For

Engineers pivoting into AI Engineering needing a flagship portfolio system capable of surviving senior panel scrutiny on table chunking, abstention, deterministic math, and eval regressions.

Human-AI Boundary

Provides one-click analyst citation override that appends failure cases directly into `eval_golden_set.jsonl` for continuous regression hardening.

Anti-Patterns to Avoid

  • Presenting only UI screenshots without before/after retrieval NDCG@5 and numerical hallucination benchmark deltas
  • Skipping negative/unanswerable query suites where a model fabricates Q4 numbers from Q3 footnotes

Latency/Cost Profile & Eval Metrics

P50 2.1s / P95 4.2s E2E with streaming citations · Reranking + generation tokens (~$0.004-0.012 per query with prompt caching)

  • Golden 10-K Benchmark Accuracy: Baseline naive chunking ~58% -> Target pipeline >= 94% (60 real 10-K comparison questions + 15 adversarial unanswerable cases)
  • Ungrounded Numerical Hallucination Rate: 0% (strict deterministic verifier pass) (Every emitted metric cross-checked against cited JSON table rows)

📦 Project Deliverables & Portfolio Artifacts

  • Docling 10-K structured parser + parent-table metadata indexer
  • Hybrid BM25/Vector + BGE-Reranker service with score-threshold abstention
  • LangGraph state machine enforcing deterministic Python financial ratio calculation node
  • 60-question benchmark report comparing Naive Chunking vs Hybrid + Rerank + Calculator

🎤 Interview Defense & System Design Deep-Dive Q&A

  1. Why use Docling + table-aware chunking plus BM25 RRF instead of purely increasing embedding context length?
  2. How do you guarantee 0% arithmetic hallucination and handle fiscal calendar mismatches across Apple/Microsoft 10-K filings?
  3. When calibrated reranker scores drop below 0.35, how does the agent recover or abstain safely without hurting UX?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

[HubB] Role Lens · Applied AI & Agent Systems Engineer[HubB] Role Lens · AI Infrastructure & LLM Inference Systems Engineer[HubB] Role Lens · Applied Scientist (Research-to-Production & Post-Training)[HubB] Role Lens · AI Decision Science, Experimentation & Eval Data Scientist[HubB] Role Lens · Forward Deployed Engineer (Enterprise AI Delivery & Solutions)[HubB] Role Lens · Machine Learning & Inference Systems Engineer[HubB] Role Lens · AI Product Manager & Applied Strategy Lead[HubB] Role Lens · Backend / Full-Stack SWE Pivoting into AI Systems[L0] Docling Layout & Complex Table PDF Parser[L0 Card] BGE-Reranker Series[L0 Card] Hybrid Search (BM25 + Dense) & RRF[L0 Card] LangGraph State Machine[L0 Card] RAG Triad Evaluation (Ragas)[L1] Complex Long-PDF Layout Parsing & Hybrid Reranked Retrieval Pattern[L2] SEC 10-K Verifiable Citation & Financial Analysis Copilot Playbook

Internal Tools & Primary Sources