Harrison Chase(@hwchase17)

building evals is hard! we're working on some skills to try to automate as much as possible. still ...

8.5内容质量
building evals is hard! we're working on some skills to try to automate as much as possible. still ...

TL;DR · AI 摘要

Harrison Chase 推出 Eval Engineering Skill，通过 Harbor 工具自动化构建评估流程，但仍需人工参与。

核心要点

  • 使用 Harbor 工具可自动化生成评估脚本，减少重复劳动
  • 评估构建流程包含代码库分析、用户反馈迭代、结果验证三阶段
  • 人工参与仍是评估优化的关键环节，自动化仅能处理基础工作

结构提纲

按章节快速跳转。

  1. §引言

    揭示评估构建的复杂性及自动化需求

  2. ·核心机制

    Eval Engineering Skill 通过代码库分析自动生成评估框架

  3. ›实施流程

    包含用户反馈迭代、Harbor 工具集成、结果验证三个阶段

  4. ·工具介绍

    Harbor 工具实现评估脚本的自动化生成与执行

  5. ›人工参与

    人工迭代是确保评估质量的关键环节

思维导图

用一张图看清主题之间的关系。

查看大纲文本（无障碍 / 无 JS 友好）
  • 自动化评估构建
    • 核心流程
      • 代码库分析
      • 用户反馈迭代
      • Harbor 工具执行
    • 关键工具
      • Harbor 自动化平台
    • 人工参与
      • 结果验证
      • 方向调整

金句 / Highlights

值得收藏与分享的关键句。

#AI#自动化#评估工程#Harbor
打开原文

Harrison Chase on X: "building evals is hard! we're working on some skills to try to automate as much as possible. still requires human in the loop, but should help bootstrap overall flow is: - give coding agent the codebase + actual traces - iterate on eval direction with user - build evals (using harbor) - run evals - look at results, iterate with user again, repeat" / X

Harrison Chase

@hwchase17

building evals is hard! we're working on some skills to try to automate as much as possible. still requires human in the loop, but should help bootstrap overall flow is: - give coding agent the codebase + actual traces - iterate on eval direction with user - build evals (using harbor) - run evals - look at results, iterate with user again, repeat

Viv

@Vtrivedy10

Jul 22

Article

Towards Automating Eval Engineering

Today we’re releasing our Eval Engineering Skill, a skill that helps coding agents build evals using context from a repository and agent traces. The skill inspects how an agent is structured, mines...

7:28 PM · Jul 22, 2026

23K

Views

23

21

201

216