trajectory labeling is really several questions, not one pass/fail - nice framing here that's how J...
TL;DR · AI 摘要
轨迹标注需多维评估而非单一判断，Edith-1模型以1/274成本超越Sonnet-5.5，LangSmith框架实现难度与正确性分离评估。
核心要点
- 轨迹标注需同时评估难度与正确性两个维度
- Edith-1模型成本仅为Sonnet-5.5的1/274
- LangSmith框架通过类型化答案提升评估精度
结构提纲
按章节快速跳转。
思维导图
用一张图看清主题之间的关系。
查看大纲文本（无障碍 / 无 JS 友好）
- 轨迹标注评估体系
- 多维评估框架
- 难度评估
- 正确性评估
- Edith-1模型
- 低成本优势
- 高准确率
- LangSmith实现
- 类型化答案系统
金句 / Highlights
值得收藏与分享的关键句。
Edith-1模型在保持平衡准确率的同时，成本仅为Sonnet-5.5的1/274
轨迹标注需同时评估任务难度与解决方案正确性两个独立维度
Jev评判系统通过类型化答案实现单次评估覆盖多维指标
Harrison Chase on X: "trajectory labeling is really several questions, not one pass/fail - nice framing here that's how Jev as a judge works in LangSmith evals: difficulty and correctness each get their own typed answer, in one pass, on every trace https://t.co/jlD2ywtlOT https://t.co/QgQpQFpZ26" / X
@hwchase17
trajectory labeling is really several questions, not one pass/fail - nice framing here that's how Jev as a judge works in LangSmith evals: difficulty and correctness each get their own typed answer, in one pass, on every trace
langchain.com/blog/jev-is-no…
Vaibhav Tulsyan
@xennygrimmato_
12h
This is a very clever usecase for Jev-like decision models. Trajectory labelling at scale is key for RSI. Why is that? Every trajectory needs to be assessed across several dimensions. 1. How difficult was the task? 2. Was the agent’s solution correct? 3. Did the agent consider a
Show more
Quote
Brando Magnani
@teknopawn
13h
Introducing edith-1, a decision model for online trace monitoring. We trained it to catch agent failures, reward hacking, and problems with tasks, graders and environments. On our eval, it beat Sonnet-5.5 in Claude Code on balanced accuracy, at an estimated 1/274 of the cost.
7:06 AM · Oct 9, 2026
·
3,019
Views
20
1
23
16