🤖 AI EngineeringID:
swe-bench-agent-evalSWE-bench Agent Evaluation
SWE-bench 代码与 Agent 评估
🎯Core Definition
SWE-bench (and SWE-bench Verified) is the gold-standard benchmark evaluating autonomous AI Coding Agents (Claude 3.5 Sonnet, Devin, Cursor Agent) on resolving real-world GitHub issues across complex multi-file codebases; curated from 2,294 (500 in Verified subset) authentic issues across flagship Python repositories (Django, SymPy, scikit-learn), it spins up isolated Docker sandboxes where an Agent must navigate directories, localize bugs, edit source files, and generate a Git patch that successfully passes repository unit-test suites.
💡Use Cases
Benchmarking Autonomous Software Engineering Agents, IDE coding copilots, and enterprise repository debugging bots.
⚡Key Problems Solved
Toy benchmarks (HumanEval) evaluate isolated 10-line single-function puzzles, detached from real-world software engineering across multi-file architectures; SWE-bench tests true repository-scale code navigation, patch synthesis, and test regression compliance.
🎯5 High-Frequency Exam Points
1
Detail the SWE-bench execution pipeline: Issue prompt -> Docker sandbox exploration -> Patch extraction -> Running unit test suites?
2
Why does SWE-bench Verified provide significantly higher evaluation fidelity than raw SWE-bench by filtering ambiguous issues?
3
How do agentic search strategies (ripgrep, ctags, AST indexing) dictate issue localization and ultimate solve rates?
4
Explain the dual verification assertions in SWE-bench: FAIL_TO_PASS (fixing the bug) and PASS_TO_PASS (preventing regression)?
5
How does transitioning to a Multi-Agent architecture (Localization -> Fix -> Test Generation) substantially boost SWE-bench resolution rates?
📖 In-depth Guide:📄 llm-as-a-judge →
Updated 2026-08-14🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "SWE-bench Agent Evaluation"
Single choice pitfall questions with instant feedback and mistake tracking.