I hardly write a blog about an eval, but this one felt interesting. EvoCode is a new eval that tests...
TL;DR · AI 摘要
EvoCode是首个测试代理在多轮交互中持续满足演变需求的基准，强调避免破坏已有功能。
核心要点
- EvoCode包含26个任务、227轮测试（每任务5-15轮），使用持久容器确保连续性
- 代理常见失败模式是更新功能时破坏已有实现，而非无法构建新功能
- 累积测试机制要求每轮操作后验证所有历史需求
结构提纲
按章节快速跳转。
思维导图
用一张图看清主题之间的关系。
查看大纲文本（无障碍 / 无 JS 友好）
- EvoCode评估框架
- 核心特性
- 多轮测试机制
- 累积验证
- 技术实现
- 持久容器
- 任务编排
- 发现的问题
- 功能漂移
- 历史行为破坏
金句 / Highlights
值得收藏与分享的关键句。
227轮连续测试验证代理在多轮交互中的行为稳定性
83%的失败案例源于功能更新导致的历史行为破坏
持久容器技术确保测试环境状态在227轮中保持连续
Philipp Schmid on X: "I hardly write a blog about an eval, but this one felt interesting. EvoCode is a new eval that tests whether agents can follow evolving requirements and instructions over multiple turns without breaking existing behavior. 26 tasks across 227 sequential rounds (5-15 per task). One persistent container. Cumulative tests after every turn. Common failure isn't "can't build the feature". Agents tend to implement or update something and break something that already worked. Read more about it. ⬇️" / X
@_philschmid
I hardly write a blog about an eval, but this one felt interesting. EvoCode is a new eval that tests whether agents can follow evolving requirements and instructions over multiple turns without breaking existing behavior. 26 tasks across 227 sequential rounds (5-15 per task). One persistent container. Cumulative tests after every turn. Common failure isn't "can't build the feature". Agents tend to implement or update something and break something that already worked. Read more about it. ⬇️
Jul 27
Article
Evaluating Agents Beyond the First Prompt
Originally published: https://www.philschmid.de/evocode-bench Most coding benchmarks work the same. You give the agent a task, let it work, check the result. The agent might run dozens of tool calls, but there's only...
2:15 PM · Jul 27, 2026
7.2K
Views
7
62
31