L0S3 · Model & AgentS5 · Eval & LLMOpsVerified: 2026-11-17
Agent Trajectory, Tool-Call & Side-Effect Failure Taxonomy Primitive
End-to-end agent evaluation primitive decomposing multi-step traces across tool selection precision, argument schema fidelity, cycle/retry bloat, and sandboxed state assertions rather than final-text grading alone.
Best For
Diagnosing flaky multi-step agents that burn redundant tokens, hallucinate tool arguments, or mutate backend state despite superficially plausible summaries.
Human-AI Boundary
Engineers bucket recurring trace failures into fixed error taxonomy codes (wrong tool, schema error, stale arg, cycle timeout, unsafe side-effect) to guide targeted fixes.
Anti-Patterns to Avoid
- Grading agents solely on final natural-language answer similarity when intermediate tool calls failed or modified wrong database rows
- Evaluating nondeterministic tool agents with a single pass@1 run instead of tracking pass^k (consistency across multiple trials)
Latency/Cost Profile & Eval Metrics
Offline mock-sandbox suite runs in seconds; full stateful DB sandbox evals in CI nightly · Multi-turn agent token amplification on retry loops
- pass^4 Multi-Trial Consistency: >= 85% deterministic task success across 4 repeated trials (Exposes brittle reasoning paths hidden by lucky single-run pass@1 scores)
- Step Overhead Ratio (Actual / Optimal Steps): <= 1.35x optimal minimal trajectory length (Flags silent tool-retry loops and malformed parameter backtracks)
💰 Business ROI Model & Evidence Level
production_practice📊 Benchmark & Alternative Comparison
- State-Diff + Trajectory Assertions (TAU-bench style): validates real backend effects and rule compliance.
- Single-turn BFCL AST Match: fast regression signal for isolated function signature and parameter extraction accuracy.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
Bidirectional Taxonomy Links
[L1] Stage 1 · Shadow-Mode AI Baseline Validation Pattern Before Live Automation[L1] Stage 3 · Structured Output Constrained FSM Decoding, Reasoning Protection & Retry Mode[L1] Stage 3 · Subagent Map-Reduce Context Isolation & Anti-Rot Pattern[L1] Stage 4 · High-Stakes Interrupt Checkpoint & Diff-Approval HITL UX Pattern[L1] Stage 5 · Production Trace-to-Golden-Dataset Failure Flywheel Pattern[L2] L2 System Design · Cloud Coding Agent Ephemeral Sandbox & Snapshot Architecture[L2] L2 Industry Playbook · Software Engineering, Code Agents & SDLC Automation AI[L2] L2 Scenario · On-Call SRE Telemetry Triage & Root-Cause Diagnostics Agent[L2] L2 Product Teardown · Claude Code Agent Harness, Tool Simplicity & Prefix Caching[L3] Capstone: Stateful On-Call SRE Telemetry Triage & Root-Cause Copilot[L0 Card] ReAct, Plan-Execute & Reflexion[L0 Card] ReAct Thought-Action-Observation[L0 Card] Agent & Tool Calling[L0 Card] SWE-bench Agent Evaluation
Internal Tools & Primary Sources
- TAU-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (paper, 2026-11-17)
- Berkeley Function Calling Leaderboard (BFCL) Evaluation Methodology (benchmark_report, 2026-11-17)