⚡Flashcard Drill➔
L0S3 · Model & AgentS5 · Eval & LLMOpsVerified: 2026-11-17

Agent Trajectory, Tool-Call & Side-Effect Failure Taxonomy Primitive

End-to-end agent evaluation primitive decomposing multi-step traces across tool selection precision, argument schema fidelity, cycle/retry bloat, and sandboxed state assertions rather than final-text grading alone.

Best For

Diagnosing flaky multi-step agents that burn redundant tokens, hallucinate tool arguments, or mutate backend state despite superficially plausible summaries.

Human-AI Boundary

Engineers bucket recurring trace failures into fixed error taxonomy codes (wrong tool, schema error, stale arg, cycle timeout, unsafe side-effect) to guide targeted fixes.

Anti-Patterns to Avoid

  • Grading agents solely on final natural-language answer similarity when intermediate tool calls failed or modified wrong database rows
  • Evaluating nondeterministic tool agents with a single pass@1 run instead of tracking pass^k (consistency across multiple trials)

Latency/Cost Profile & Eval Metrics

Offline mock-sandbox suite runs in seconds; full stateful DB sandbox evals in CI nightly · Multi-turn agent token amplification on retry loops

  • pass^4 Multi-Trial Consistency: >= 85% deterministic task success across 4 repeated trials (Exposes brittle reasoning paths hidden by lucky single-run pass@1 scores)
  • Step Overhead Ratio (Actual / Optimal Steps): <= 1.35x optimal minimal trajectory length (Flags silent tool-retry loops and malformed parameter backtracks)

💰 Business ROI Model & Evidence Level

production_practice

📊 Benchmark & Alternative Comparison

  • State-Diff + Trajectory Assertions (TAU-bench style): validates real backend effects and rule compliance.
  • Single-turn BFCL AST Match: fast regression signal for isolated function signature and parameter extraction accuracy.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔

Bidirectional Taxonomy Links

[L1] Stage 1 · Shadow-Mode AI Baseline Validation Pattern Before Live Automation[L1] Stage 3 · Structured Output Constrained FSM Decoding, Reasoning Protection & Retry Mode[L1] Stage 3 · Subagent Map-Reduce Context Isolation & Anti-Rot Pattern[L1] Stage 4 · High-Stakes Interrupt Checkpoint & Diff-Approval HITL UX Pattern[L1] Stage 5 · Production Trace-to-Golden-Dataset Failure Flywheel Pattern[L2] L2 System Design · Cloud Coding Agent Ephemeral Sandbox & Snapshot Architecture[L2] L2 Industry Playbook · Software Engineering, Code Agents & SDLC Automation AI[L2] L2 Scenario · On-Call SRE Telemetry Triage & Root-Cause Diagnostics Agent[L2] L2 Product Teardown · Claude Code Agent Harness, Tool Simplicity & Prefix Caching[L3] Capstone: Stateful On-Call SRE Telemetry Triage & Root-Cause Copilot[L0 Card] ReAct, Plan-Execute & Reflexion[L0 Card] ReAct Thought-Action-Observation[L0 Card] Agent & Tool Calling[L0 Card] SWE-bench Agent Evaluation

Internal Tools & Primary Sources