How to Find the Agent Failures Your Evals Miss with Scott Clark

Q: 日志记录

用于收集系统的运行时数据。

Q: 已知信号监控

用于监测已知的问题信号。

Q: 在线分析

用于发现未知的未知问题。

TWIML AI Podcast

TWIML AI Podcast播客2026年5月7日53:19

How to Find the Agent Failures Your Evals Miss with Scott Clark - #767

7.5内容质量

播客收听

时长 53:19原播客页面

问这期播客

会先在本集摘要、章节、转录和笔记里找答案。

TL;DR · AI 摘要

Scott Clark讨论了如何通过层次化的可观测性方法发现复杂LLM系统中的未检测到的代理失败，强调了在线分析和自适应方法的重要性。

核心要点

使用层次化的可观测性方法发现未知问题
在线分析和自适应方法对非稳定模型至关重要
OpenTelemetry和GenAI语义规范有助于系统监控

结构提纲

按章节快速跳转。

§关于本集
Scott Clark讨论了如何通过层次化的可观测性方法发现复杂LLM系统中的未检测到的代理失败。
·层次化的可观测性
包括日志记录、已知信号监控和在线分析，以发现未知问题。
›日志记录
用于收集系统的运行时数据。
›已知信号监控
用于监测已知的问题信号。
›在线分析
用于发现未知的未知问题。
·在线分析和自适应方法
对于非稳定模型至关重要，能够生成评估、护栏和训练数据。

思维导图

用一张图看清主题之间的关系。

查看大纲文本（无障碍 / 无 JS 友好）

发现LLM系统中的代理失败

金句 / Highlights

值得收藏与分享的关键句。

Scott介绍了一种Maslow的层次化的可观测性方法，包括日志记录、监控和在线分析。
— 第2段
⬇︎ 下载 PNG 𝕏 分享到 X
在线分析能够发现标准评估遗漏的代理失败，如'懒惰'工具使用的幻觉。
— 第2段
⬇︎ 下载 PNG 𝕏 分享到 X
在线分析能够生成评估、护栏和训练数据，从而形成数据飞轮效应。
— 第2段
⬇︎ 下载 PNG 𝕏 分享到 X

章节

要点
使用层次化的可观测性方法发现未知问题
使用层次化的可观测性方法发现未知问题
要点
在线分析和自适应方法对非稳定模型至关重要
在线分析和自适应方法对非稳定模型至关重要
要点
OpenTelemetry和GenAI语义规范有助于系统监控
OpenTelemetry和GenAI语义规范有助于系统监控

转录

这期还没有可搜索转录。后续抓到带时间戳的内容后会自动补到这里。

#LLM#可观测性#机器学习#代理失败

节目笔记

How to Find the Agent Failures Your Evals Miss | TWIML - The Voice of Machine Learning & AI

About Contact Newsletter

Twiml icon youtube Twiml icon X/twitter Twiml icon linkedin Twiml icon Facebook Twiml icon instagram

[](https://twimlai.com/)

EPISODE 767

|

MAY 7, 2026

Watch

Play

Follow

![Image 1Apple Podcasts](https://podcasts.apple.com/us/podcast/the-twiml-ai-podcast-formerly-this-week-in-machine/id1116303051)

![Image 2Spotify](https://open.spotify.com/show/2sp5EL7s7EqxttxwwoJ3i7)

![Image 3YouTube](https://www.youtube.com/twimlai)

![Image 4Overcast](https://overcast.fm/itunes1116303051)

![Image 5Podcast Addict](https://podcastaddict.com/podcast/2990273)

![Image 6Castbox](https://castbox.fm/channel/id4477675)

![Image 7Pocket Casts](https://pocketcasts.com/podcast/the-twiml-ai-podcast-formerly-this-week-in-machine-learning-artificial-intelligence/8fc91c30-03a7-0134-9c92-59d98c6b72b8)

![Image 8RSS](https://feeds.megaphone.fm/MLN2155636147)

Share

![Image 9LinkedIn](https://twimlai.com/podcast/twimlai/how-find-agent-failures-your-evals-miss)

![Image 10X](https://twimlai.com/podcast/twimlai/how-find-agent-failures-your-evals-miss)

![Image 11Facebook](https://twimlai.com/podcast/twimlai/how-find-agent-failures-your-evals-miss)

![Image 12Reddit](https://twimlai.com/podcast/twimlai/how-find-agent-failures-your-evals-miss)

Don't Miss an Episode!_Join our mailing list for episode summaries and other updates._

Don't fill this out if you're human:

First Name

Last Name

Email

JOIN LIST

About this Episode

In this episode, Scott Clark, co-founder and CEO of Distributional, joins us to explore how teams can reliably operate and improve complex LLM systems and agents in production. Scott introduces a Maslow’s hierarchy of observability: telemetry for logging, monitoring for known signals, and post-production or online analytics to surface unknown unknowns. We dig into examples of real-world failures Scott’s team has seen in production systems, such as “lazy” tool-use hallucinations that standard evals miss, and how mapping traces into vector fingerprints enables clustering and topic discovery to uncover emergent behaviors. Scott explains how analytics can feed the data flywheel by generating evals, guardrails, and training data, and why online, adaptive approaches are essential for non-stationary models. We also touch on practical how-to’s such as instrumentation with OpenTelemetry, the GenAI semantic conventions, and the role of dedicated analytics tools.

About the Guest

#### Scott Clark Distributional

Connect with Scott

Thanks to our sponsor Distributional

This show is brought to you by our friends at Distributional, the AI analytics platform built for teams that are serious about agent quality. Distributional finds patterns in production agent traces, creating actionable insights with suggestions for new evals, refined guardrails, and improvements to your agent based on real usage. Don't take their word for it, use Distributional yourself for free. Go to app.dbnl.com to create your free hosted account that also comes with a free LLM endpoint to power evals and analytics. Or install a self-hosted version of Distributional for free in your own environment. To learn more, visit dbnl.com and start improving production agent quality today.