L3S2 · Data & RetrievalS3 · Model & AgentS4 · UX & HITLS5 · Eval & LLMOpsS6 · Security & FinOpsVerified: 2026-11-15
Verifiable-Citation SEC 10-K Research Copilot (Reference Portfolio Project)
High-signal portfolio reference implementation combining Docling filing ingestion, BM25 + dense RRF + BGE-Reranker retrieval, LangGraph deterministic math verification, and a 60-query numerical faithfulness CI eval suite.
Best For
Engineers pivoting into AI Engineering needing a flagship portfolio system capable of surviving senior panel scrutiny on table chunking, abstention, deterministic math, and eval regressions.
Human-AI Boundary
Provides one-click analyst citation override that appends failure cases directly into `eval_golden_set.jsonl` for continuous regression hardening.
Anti-Patterns to Avoid
- Presenting only UI screenshots without before/after retrieval NDCG@5 and numerical hallucination benchmark deltas
- Skipping negative/unanswerable query suites where a model fabricates Q4 numbers from Q3 footnotes
Latency/Cost Profile & Eval Metrics
P50 2.1s / P95 4.2s E2E with streaming citations · Reranking + generation tokens (~$0.004-0.012 per query with prompt caching)
- Golden 10-K Benchmark Accuracy: Baseline naive chunking ~58% -> Target pipeline >= 94% (60 real 10-K comparison questions + 15 adversarial unanswerable cases)
- Ungrounded Numerical Hallucination Rate: 0% (strict deterministic verifier pass) (Every emitted metric cross-checked against cited JSON table rows)
📦 Project Deliverables & Portfolio Artifacts
- Docling 10-K structured parser + parent-table metadata indexer
- Hybrid BM25/Vector + BGE-Reranker service with score-threshold abstention
- LangGraph state machine enforcing deterministic Python financial ratio calculation node
- 60-question benchmark report comparing Naive Chunking vs Hybrid + Rerank + Calculator
🎤 Interview Defense & System Design Deep-Dive Q&A
- Why use Docling + table-aware chunking plus BM25 RRF instead of purely increasing embedding context length?
- How do you guarantee 0% arithmetic hallucination and handle fiscal calendar mismatches across Apple/Microsoft 10-K filings?
- When calibrated reranker scores drop below 0.35, how does the agent recover or abstain safely without hurting UX?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
Bidirectional Taxonomy Links
[HubB] Role Lens · Applied AI & Agent Systems Engineer[HubB] Role Lens · AI Infrastructure & LLM Inference Systems Engineer[HubB] Role Lens · Applied Scientist (Research-to-Production & Post-Training)[HubB] Role Lens · AI Decision Science, Experimentation & Eval Data Scientist[HubB] Role Lens · Forward Deployed Engineer (Enterprise AI Delivery & Solutions)[HubB] Role Lens · Machine Learning & Inference Systems Engineer[HubB] Role Lens · AI Product Manager & Applied Strategy Lead[HubB] Role Lens · Backend / Full-Stack SWE Pivoting into AI Systems[L0] Docling Layout & Complex Table PDF Parser[L0 Card] BGE-Reranker Series[L0 Card] Hybrid Search (BM25 + Dense) & RRF[L0 Card] LangGraph State Machine[L0 Card] RAG Triad Evaluation (Ragas)[L1] Complex Long-PDF Layout Parsing & Hybrid Reranked Retrieval Pattern[L2] SEC 10-K Verifiable Citation & Financial Analysis Copilot Playbook
Internal Tools & Primary Sources
- Docling Open-Source Document Parser (github, 2026-11-15)
- LangGraph Agent State Machine Guide (official_docs, 2026-11-15)