Microsoft Research Blog

Systematic debugging for AI agents: Introducing the AgentRx framework

2.5内容质量
Systematic debugging for AI agents: Introducing the AgentRx framework

TL;DR · AI 摘要

微软研究院提出AgentRx框架，旨在为AI智能体提供系统化的调试方法与工具链，以解决复杂Agent行为不可控、错误难追踪等工程痛点。但提供的正文仅为网站导航栏，缺乏具体架构、实现原理与实践案例等核心技术细节。

核心要点

  • 微软研究院发布AgentRx框架，填补AI智能体系统化调试工具空白。
  • 该框架提供标准化调试流程，旨在提升复杂Agent行为的可观测性与排错效率。
  • 提供的正文内容缺失，仅为网站导航结构，无法评估具体技术实现与工程价值。
#AI Agent#调试框架#Microsoft Research#可观测性#软件工程
打开原文

Systematic debugging for AI agents: Introducing the AgentRx framework - Microsoft Research

Skip to main content

Image 4: Microsoft
Image 4: Microsoft

Research

Our research

#### Resources

#### Research areas: Intelligence

#### Research areas: Systems

#### Research areas: Theory

#### Research areas: Other Sciences

Programs & events

Academic programsEvents & academic conferencesMicrosoft Research Forum

Connect & learn

Behind the Tech podcastMicrosoft Research blogMicrosoft Research ForumMicrosoft Research podcast

About

#### People & news

#### Microsoft Research Labs

#### Other labs

More

Register: Research Forum

All Microsoft

#### Tech & innovation

#### Industries

#### Partners

#### Resources

View Sitemap

Cancel

Search

Return to Blog Home

Microsoft Research Blog

Systematic debugging for AI agents: Introducing the AgentRx framework

Published March 12, 2026

By Shraddha Barke, Senior Researcher Arnav Goyal, Research Fellow Alind Khare, Senior Researcher Chetan Bansal, Senior Principal Research Manager

Share this page

  • [Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fblog%2Fsystematic-debugging-for-ai-agents-introducing-the-agentrx-framework%2F "Share on Facebook")
  • [Share on X](https://x.com/intent/tweet?text=Systematic%20debugging%20for%20AI%20agents%3A%20Introducing%20the%20AgentRx%20framework&url=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fblog%2Fsystematic-debugging-for-ai-agents-introducing-the-agentrx-framework%2F "Share on X")
  • [Share on LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fblog%2Fsystematic-debugging-for-ai-agents-introducing-the-agentrx-framework%2F&title=Systematic%20debugging%20for%20AI%20agents%3A%20Introducing%20the%20AgentRx%20framework&summary=Systematic%20debugging%20for%20AI%20agents%3A%20Introducing%20the%20AgentRx%20framework&source=Microsoft%20Research "Share on LinkedIn")
  • [Share on Reddit](http://www.reddit.com/submit?title=Systematic%20debugging%20for%20AI%20agents%3A%20Introducing%20the%20AgentRx%20framework&url=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fresearch%2Fblog%2Fsystematic-debugging-for-ai-agents-introducing-the-agentrx-framework%2F "Share on Reddit")
  • [Subscribe to our RSS feed](https://www.microsoft.com/en-us/research/feed/ "Subscribe to our RSS feed")
Image 5: Three white line icons, showing network, workflow, and bug‑analysis icons, on a blue‑to‑purple gradient background.
Image 5: Three white line icons, showing network, workflow, and bug‑analysis icons, on a blue‑to‑purple gradient background.

At a glance

  • Problem: Debugging AI agent failures is hard because trajectories are long, stochastic, and often multi-agent, so the true root cause gets buried.
  • Solution:[AgentRx (opens in new tab)](https://aka.ms/AgentRx/Repo) pinpoints the _first unrecoverable (“critical failure”) step_ by synthesizing guarded, executable constraints from tool schemas and domain policies, then logging evidence-backed violations step-by-step.
  • Benchmark + taxonomy: We release [AgentRx Benchmark (opens in new tab)](https://aka.ms/AgentRx/Dataset) with 115 manually annotated failed trajectories across τ-bench, Flash, and Magentic-One, plus a grounded nine-category failure taxonomy.
  • Results + release: AgentRx improves failure localization (+23.6%) and root-cause attribution (+22.9%) over prompting baselines, and we are open-sourcing the framework and dataset.

As AI agents transition from simple chatbots to autonomous systems capable of managing cloud incidents, navigating complex web interfaces, and executing multi-step API workflows, a new challenge has emerged: transparency.

When a human makes a mistake, we can usually trace the logic. But when an AI agent fails, perhaps by hallucinating a tool output or deviating from a security policy ten steps into a fifty-step task, identifying exactly where and why things went wrong is an arduous, manual process.

Today, we are excited to announce the open-source release of **AgentRx** (opens in new tab), an automated, domain-agnostic framework designed to pinpoint the “critical failure step” in agent trajectories. Alongside the framework, we are releasing the [AgentRx Benchmark (opens in new tab)](https://aka.ms/AgentRx/Dataset), a dataset of 115 manually annotated failed trajectories to help the community build more transparent, resilient agentic systems.

The challenge: Why AI agents are hard to debug

Modern AI agents are often:

  • Long-horizon: They perform dozens of actions over extended periods.
  • Probabilistic: The same input might lead to different outputs, making reproduction difficult.
  • Multi-agent: Failures can be “passed” between agents, masking the original root cause.

Traditional success metrics (like “Did the task finish?”) don’t tell us enough. To build safe agents, we need to identify the exact moment a trajectory becomes unrecoverable and capture evidence for what went wrong at that step.

Introducing AgentRx: An automated diagnostic “prescription”

AgentRx (short for “Agent Diagnosis”) treats agent execution like a system trace that needs validation. Instead of relying on a single LLM to “guess” the error, AgentRx uses a structured, multi-stage pipeline:

  1. Trajectory normalization: Heterogeneous logs from different domains are converted into a common intermediate representation.
  2. Constraint synthesis: The framework automatically generates executable constraints based on tool schemas (e.g., “The API must return a valid JSON response”) and domain policies (e.g., “Do not delete data without user confirmation”).
  3. Guarded evaluation: AgentRx evaluates constraints step-by-step, checking each constraint only when its _guard condition_ applies, and produces an auditable validation log of evidence-backed violations.
  4. LLM-based judging: Finally, an LLM judge uses the validation log and a grounded failure taxonomy to identify the Critical Failure Step—the first unrecoverable error.
Image 6: Flowchart illustrating an agent failure attribution pipeline. In the upper left, a blue rounded box labeled “Task Context” contains three stacked inputs: “Domain Policy,” “Tool Schema,” and “Trajectory.” A downward arrow leads into a large yellow rounded rectangle representing the validation pipeline. Inside this area, a green box labeled “Constraint Generator” feeds into a blue box labeled “Constraint Checker.” To their right is a JSON-like constraint specification with fields such as assertion_name:
Image 6: Flowchart illustrating an agent failure attribution pipeline. In the upper left, a blue rounded box labeled “Task Context” contains three stacked inputs: “Domain Policy,” “Tool Schema,” and “Trajectory.” A downward arrow leads into a large yellow rounded rectangle representing the validation pipeline. Inside this area, a green box labeled “Constraint Generator” feeds into a blue box labeled “Constraint Checker.” To their right is a JSON-like constraint specification with fields such as assertion_name:

_The AgentRx workflow:_ Given a failed trajectory, tool schemas, and domain policy, AgentRx synthesizes guarded constraints, evaluates them step-by-step to produce an auditable violation log with evidence, and uses an LLM judge to predict the critical failure step and root-cause category.

A New Benchmark for Agent Failures

To evaluate AgentRx, we developed a manually annotated benchmark consisting of 115 failed trajectories across three complex domains:

  • τ-bench: Structured API workflows for retail and service tasks.
  • Flash: Real-world incident management and system troubleshooting.
  • Magentic-One: Open-ended web and file tasks using a generalist multi-agent system.

Using a grounded-theory approach, we derived a nine-category failure taxonomy that generalizes across these domains. This taxonomy helps developers distinguish between a “Plan Adherence Failure” (where the agent ignored its own steps) and an “Invention of New Information” (hallucination).

| Taxonomy Category | Description | | --- | --- | | Plan Adherence Failure | Ignored required steps / did extra unplanned actions | | Invention of New Information | Altered facts not grounded in trace/tool output | | Invalid Invocation | Tool call malformed / missing args / schema-invalid | | Misinterpretation of Tool Output | Read tool output incorrectly; acted on wrong assumptions | | Intent–Plan Misalignment | Misread user goal/constraints and planned wrongly | | Under-specified User Intent | Could not proceed because required info wasn’t available | | Intent Not Supported | No available tool can do what’s being asked | | Guardrails Triggered | Execution blocked by safety/access restrictions | | System Failure | Connectivity/tool endpoint failures |

Image 7: Two-column taxonomy table with a dark blue header row labeled “Taxonomy Category” and “Description.” The rows define nine agent failure types: Plan Adherence Failure, Invention of New Information, Invalid Invocation, Misinterpretation of Tool Output, Intent–Plan Misalignment, Under-specified User Intent, Intent Not Supported, Guardrails Triggered, and System Failure. Their descriptions explain, respectively, skipped or extra actions, invented facts, malformed tool calls, incorrect reading of tool outputs, wrong planning from misunderstood intent, inability to proceed due to missing information, lack of tool support, blocking by safety or access controls, and connectivity or endpoint failures.
Image 7: Two-column taxonomy table with a dark blue header row labeled “Taxonomy Category” and “Description.” The rows define nine agent failure types: Plan Adherence Failure, Invention of New Information, Invalid Invocation, Misinterpretation of Tool Output, Intent–Plan Misalignment, Under-specified User Intent, Intent Not Supported, Guardrails Triggered, and System Failure. Their descriptions explain, respectively, skipped or extra actions, invented facts, malformed tool calls, incorrect reading of tool outputs, wrong planning from misunderstood intent, inability to proceed due to missing information, lack of tool support, blocking by safety or access controls, and connectivity or endpoint failures.

_Analysis of failure density across domains. In multi-agent systems like Magentic-One, trajectories often contain multiple errors, but AgentRx focuses on identifying the first critical breach._

Key Results

In our experiments, AgentRx demonstrated significant improvements over existing LLM-based prompting baselines:

  • +23.6% absolute improvement in failure localization accuracy.
  • +22.9% improvement in root-cause attribution.

By providing the “why” behind a failure through an auditable log, AgentRx allows developers to move beyond trial-and-error prompting and toward systematic agentic engineering.

Join the Community: Open Source Release

We believe that agent reliability is a prerequisite for real-world deployment. To support this, we are open sourcing the AgentRx framework and the complete annotated benchmark.

We invite researchers and developers to use AgentRx to diagnose their own agentic workflows and contribute to the growing library of failure constraints. Together, we can build AI agents that are not just powerful, but auditable, and reliable.

Acknowledgements

We would like to thank Avaljot Singh and Suman Nath for contributing to this project.

Opens in a new tab

Related publications

[AgentRx: Diagnosing AI Agent Failures from Execution Trajectories](https://www.microsoft.com/en-us/research/publication/agentrx-diagnosing-ai-agent-failures-from-execution-trajectories/)

Meet the authors

Image 8: Portrait of Shraddha Barke
Image 8: Portrait of Shraddha Barke

Shraddha Barke

Senior Researcher

Learn more

Image 9: Portrait of Arnav Goyal
Image 9: Portrait of Arnav Goyal

Arnav Goyal

Research Fellow

Learn more

Image 10: Portrait of Alind Khare
Image 10: Portrait of Alind Khare

Alind Khare

Senior Researcher

Learn more

Image 11: Portrait of Chetan Bansal
Image 11: Portrait of Chetan Bansal

Chetan Bansal

Senior Principal Research Manager

Learn more

Continue reading

Image 12: Three white line icons on a blue-to-purple gradient background: the first icon shows a simple flowchart with connected squares and a diamond, the second icon depicts a network of interconnected circles, and the third icon represents a social connection diagram with three user profile symbols linked together.
Image 12: Three white line icons on a blue-to-purple gradient background: the first icon shows a simple flowchart with connected squares and a diamond, the second icon depicts a network of interconnected circles, and the third icon represents a social connection diagram with three user profile symbols linked together.

December 11, 2025

[Agent Lightning: Adding reinforcement learning to AI agents without code rewrites](https://www.microsoft.com/en-us/research/blog/agent-lightning-adding-reinforcement-learning-to-ai-agents-without-code-rewrites/)

Image 13: This figure denotes a human figure above a small monitor on the left and a gear on the right with arrows pointing to each.
Image 13: This figure denotes a human figure above a small monitor on the left and a gear on the right with arrows pointing to each.

May 19, 2025

[Magentic-UI, an experimental human-centered web agent](https://www.microsoft.com/en-us/research/blog/magentic-ui-an-experimental-human-centered-web-agent/)

Image 14: DeBug blog graphic with a line drawing of a computer screen on the left and a magnifying glass with a bug on the right.
Image 14: DeBug blog graphic with a line drawing of a computer screen on the left and a magnifying glass with a bug on the right.

April 10, 2025

[Debug-gym: an environment for AI coding tools to learn how to debug code like programmers](https://www.microsoft.com/en-us/research/blog/debug-gym-an-environment-for-ai-coding-tools-to-learn-how-to-debug-code-like-programmers/)

Image 15: White outline illustrations for AIOps on a blue and green gradient background.
Image 15: White outline illustrations for AIOps on a blue and green gradient background.

December 20, 2024

[AIOpsLab: Building AI agents for autonomous clouds](https://www.microsoft.com/en-us/research/blog/aiopslab-building-ai-agents-for-autonomous-clouds/)

See all blog posts

Research Areas

  • ![Image 16 Artificial intelligence](https://www.microsoft.com/en-us/research/research-area/artificial-intelligence/)

Research Groups

Related labs

Follow us:

Share this page:

What's new

Microsoft Store

Education

Business

Developer & IT

Company

Your Privacy Choices

Consumer Health Privacy

Notifications