Cool paper from Apple. Most evaluation of tool-calling agents happens after the trajectory is over....
TL;DR · AI 摘要
Apple提出的新方法将评估嵌入到工具调用的执行循环中，显著提升了准确性，无需重新训练基础代理。
核心要点
- 新方法提升准确性，无需重训基础代理
- 帮助度和危害度指标量化改进效果
- 推理模型审查者比GPT-4o更有效
结构提纲
按章节快速跳转。
思维导图
用一张图看清主题之间的关系。
查看大纲文本（无障碍 / 无 JS 友好）
- Apple的新评估方法
金句 / Highlights
值得收藏与分享的关键句。
新方法将评估嵌入到执行循环中，显著提升了准确性，无需重新训练基础代理。
帮助度测量修复的基础错误百分比；危害度测量正确调用因审查而退化的比例。
推理模型审查者得到3:1的效益风险比，而GPT-4o仅为2.1:1。
Most evaluation of tool-calling agents happens after the trajectory is over. By then the wrong call has already shipped.
This new paper moves evaluation into the execution loop. A specialized reviewer agent inspects each provisional tool call before it https://t.co/JKCt67mfWJ" / X
Cool paper from Apple. Most evaluation of tool-calling agents happens after the trajectory is over. By then the wrong call has already shipped. This new paper moves evaluation into the execution loop. A specialized reviewer agent inspects each provisional tool call before it executes. If something is off, it injects feedback and the primary agent revises. To quantify the tradeoff between corrections and new mistakes, they introduce Helpfulness-Harmfulness metrics. Helpfulness measures the percentage of base errors fixed; harmfulness measures correct calls degraded by the reviewer. Results on BFCL: +5.5% on irrelevance detection (84.9% to 90.4%), +1.6% on relevance, all with no retraining of the base agent. On τ²-Bench multi-turn: +7.1% (48.7% to 55.8%). Reasoning-model reviewers get a 3:1 benefit-to-risk ratio vs. 2.1:1 for GPT-4o. Adding GEPA prompt optimization stacks another +1.5–2.8%. Why does it matter? You can keep the base tool-calling agent frozen and still ship measurable accuracy gains by improving only the reviewer. Model selection and prompt optimization on the reviewer become real, separable production levers. Paper: arxiv.org/abs/2604.27233 Learn to build effective AI agents in our academy: academy.dair.ai
