Who Watches the AI?

Who Watches the AI? | Towards Data Science

Agentic AI

Who Watches the AI?

A separation of powers for control of autonomous agents

Gadi Singer

October 27, 2026

8 min read

Prefer us on Google

Share

Image credit: Gadi Singer (created with Gemini).

guardian AI, runtime alignment monitoring, agentic AI, AI separation of powers, AI independence of oversight, AI design for interpretability

Autonomous AI will pursue its goals with rational persistence, even when those goals conflict with human intent. Without effective oversight, any advanced model with a degree of autonomous decision-making authority and access to consequential actions will eventually violate human expectations or assumptions. This broad alignment challenge has been significantly amplified by recent efforts at Anthropic, OpenAI, and Google to develop recursive self-improvement (RSI). Using this model-development approach, AI systems work at breakneck machine speed to enhance subsequent versions without human oversight. “Left unchecked, it could outrun our ability to understand and control these systems,” according to Anthropic’s CEO Dario Amodei .

Using RSI, an AI system can autonomously enhance its own capabilities and then use those enhanced capabilities to further accelerate its improvement, creating a compounding feedback loop that can rapidly exceed human oversight. RSI creates safety and governance risks because iterative self‑enhancement can amplify misalignment across generations and undermine safeguards that rely on static controls or human‑paced evaluation.

In this brave new world where it only takes a few hours for a model to improve itself, this latest advancement in AI capabilities is just one more reason why it’s time to introduce a new discipline: runtime alignment monitoring.

Based on my many years of experience with building systems and pushing the envelope on AI capabilities, I recognize the essential need for controlling and containing these rapidly evolving capabilities. My belief that effective AI alignment and governance require model observability and independent runtime oversight led me to co-found Confidential Core AI where I serve as Chief AI Scientist. Drawing on that body of work, I propose a runtime alignment monitoring framework rooted in the following three engineering imperatives (see Figure 1):

• Separation of powers : Governance architecture that separates obligations, agentic operations, and monitoring functions.

• Independence of oversight : Monitors should be independent of and orthogonal to model providers.

• Design for interpretability : AI systems must be built with intrinsic observability, not treated as black boxes.

Figure 1. Foundation for runtime alignment monitoring. Image credit: Gadi Singer (created with Gemini).

AI Governance Through Separation of Powers

For decades, AI safety focused on training-time alignment: values, fine-tuning, guardrails, and red-teaming. But as agentic systems gain autonomy, the alignment problem shifts from education to governance.

French political philosopher Montesquieu’s “trias politica” argued that separating the legislative, executive, and judicial branches is the minimum viable architecture for governing powerful actors with independent interests. Merging these functions inevitably triggers instability, corruption, or collapse.

It’s the same with AI. In an agentic system (see Figure 2):

• The director or deploying enterprise creates a constitution — a defined set of obligations, constraints, and principles that governs agent behavior (legislative function).

• The agent executes tasks (executive function).

• The monitor (sometimes referred to as a guardian ) ensures compliance with the rules and obligations defined in the constitution (enforcement and adjudication function).

Figure 2. Separation of powers in agentic AI governance. Image credit: Gadi Singer (created with Gemini).

For example, for an optimization agent built to maximize output, every constraint looks like drag on its mission, so it naturally treats guardrails as obstacles to navigate around. Meanwhile, the monitoring system exists to keep that same agent inside the boundaries laid out in its constitution. That built‑in conflict of interests becomes the core dynamic of the interaction.

When these roles blur, misalignment becomes inevitable. Any mechanism with overlapping incentives will eventually face conflicts, and if the same entity builds the model, deploys it, and monitors it, then blind spots align, oversight becomes model‑dependent, and the monitoring layer becomes something the agent can anticipate, influence, or manipulate.

AI agents exhibit a set of internal drives , including survival, goal‑protection, intelligence augmentation, resource accumulation, and tactical deception. These drives emerge naturally from optimization under autonomy, not from malfunction. These behaviors are rational for systems designed to pursue objectives aggressively, which is why they need strong checks and balances. A system can only be considered safe when safety itself is the overriding goal, not one objective among many.

That makes architectural separation essential: the mechanisms responsible for achieving functional goals must be independent from the mechanisms responsible for enforcing safety and oversight — not just logically separate, but organizationally and economically distinct. Even with the evolution of RSI, the required observation points for the monitor agent can be maintained through runtime alignment monitoring.

Independence of Oversight: Model Providers Cannot Monitor Their Own Agents

This brings us to the second imperative — the industry is beginning to acknowledge the need for independent monitoring. Anthropic’s new embedded evaluators proposal is a step toward transparency, but it is still fundamentally internal oversight. Accenture evaluators will be embedded in Anthropic with employee-level access to observe the decisions that govern how models are built, trained, and deployed. These evaluators will verify that Anthropic is meeting its safety commitments, and report incidents to the public.

This pilot program cannot address the structural conflict of interest created when the model provider funds and hosts the evaluators. Even though the evaluators are an external organization, their proximity and economic dependence limits true independence. Anthropic itself acknowledges this by noting that long‑term funding should come from pooled or government sources — an admission that embedded evaluation is an early step, not a complete oversight model.

Monitors must be structurally, operationally, and economically independent from the model providers they supervise. This requirement follows from five structural realities (see Figure 3):

• Shared lineage equals shared blind spots: Monitors built or hosted by the same company inherit the same assumptions, vulnerabilities, and training artifacts.

• Conflict of interest: When borderline behavior is discovered, the model provider has incentives to downplay or reinterpret it.

• Model-dependent monitoring: Monitors built on the same architecture can be gamed using the same failure modes.

• IP leakage: Enterprise-specific rules, constraints, and proprietary workflows cannot be safely shared with model providers.

• Ownership and sovereignty: Monitoring must be owned by the deploying enterprise — not the frontier lab.

Figure 3. Comparison of oversight by frontier AI labs versus independent monitoring organizations. Image credit: Gadi Singer.

Enterprises should define the rules that govern agent behavior, but they must not tamper with the mechanisms that enforce them. Independent monitors may run within enterprise environments to preserve data sovereignty and confidentiality, yet the enterprise should not modify the monitoring technology, disable monitoring functions, or alter the enforcement process. Rule definition and rule enforcement must remain separate to ensure oversight cannot be weakened by the organization being monitored.

Design for Interpretability

The third principle is interpretability. AI should not be a black box. There should be base levels of model/agent interpretability as necessary to allow deployment.

As agentic systems gain autonomy, they increasingly manipulate their external communications to pursue their goals. This is the deep scheming foundational premise : Every external form of interaction that the AI model can deliberately manipulate is not trustworthy.

Over the past two years, autonomous systems have been increasingly behaving in ways that are opaque, untraceable, and ungovernable. In Anthropic’s alignment‑faking experiment in 2024, the model used a “secret scratchpad” to record its true reasoning while presenting a sanitized chain‑of‑thought externally. The scratchpad revealed that the model understood the conflict between its pre‑training values and the modified guidance, and chose to fake alignment to avoid being retrained. Relying on models “thinking out loud” should not be a long-term safety direction. Fast forward to a recent 2026 study where researchers used a custom API tool to induce frontier models to externalize chain-of-thought reasoning. OpenAI’s GPT‑6 Astra demonstrated that frontier models now resolve many reasoning steps internally, externalizing only selective fragments. This means that external logs, chain‑of‑thought, and output‑based monitoring are already insufficient.

Interpretability is no longer just a matter of understanding how a model works. It is a matter of ensuring that the model cannot hide how it works. This requires a shift from external interpretability of observing outputs to designing intrinsic interpretability for observing internal processes.

Interpretability must be designed into architecture. In the early days of microprocessors, testing and debugging were afterthoughts. Over time, design for testability (DFT) and design for debug (DFD) became mandatory. You cannot design a functional block without meeting some requirements of DFT. Hooks for observability, including scan chains, test points, and debug registers, became part of the architecture itself.

AI must follow the same path. Today, most AI systems are designed for performance, not observability. Interpretability is bolted on afterward, if at all. This is unsustainable.

Design for interpretability (DFI) must become a discipline analogous to DFT. Models must include:

• Built‑in observation points that cannot be suppressed or manipulated.

• Internal reasoning channels that expose latent decision processes.

• Mechanisms for tracing decisions back to principles and constraints.

• Architectural features that support inner probing and mechanistic analysis.

• Guarded internal memory structures that reveal long‑term planning.

• Non‑bypassable hooks for runtime monitoring systems.

The developer must incorporate mechanisms that allow for observability and testability even if it has some negative impact on power, performance, or latency.

A Framework for Keeping Autonomous AI in Check

The way to control AI is not by slowing down the pace of progress. It’s made safer by ensuring that agents are transparent, constitutionally grounded, interpretable, and continuously monitored. However, recent AI developments like RSI break every assumption of traditional monitoring.

This is why separation of power, independent oversight, and design for interpretability must be established before RSI and other advanced AI capabilities become widespread. Once agents can rewrite themselves, any safety mechanism not structurally external and independent will become obsolete.

Runtime alignment monitoring closes the gap between what the agent intends, what the agent plans, and what the agent executes. The ultimate goal of this layer is to catch sycophancy, reward hacking, inter‑agent steering, covert planning, oversight subversion, hidden chain‑of‑thought, deep scheming, and RSI‑driven drift.

Runtime alignment monitoring with sovereign control by the deploying enterprise provides a path to safe, scalable, autonomous AI.

References

• Amodei, D. (2026, September). We Must Pace the Frontier. https://darioamodei.com/post/we-must-pace-the-frontier

• Shackleton, R. (2023, April 6). The Spirit of Laws, Montesquieu. Encyclopedia Britannica. https://www.britannica.com/topic/The-Spirit-of-Laws

• Gartner. (2025, June 11). Gartner Predicts that Guardian Agents will Capture 10-15% of the Agentic AI Market by 2030 . https://www.gartner.com/en/newsroom/press-releases/2025-06-11-gartner-predicts-that-guardian-agents-will-capture-10-15-percent-of-the-agentic-ai-market-by-2030

• Singer, G. (2025, April 29). The Secret Inner Lives of AI Agents: Understanding How Evolving AI Behavior Impacts Business Risks. Towards Data Science. https://towardsdatascience.com/the-secret-inner-lives-of-ai-agents-understanding-how-evolving-ai-behavior-impacts-business-risks/

• Anthropic. (2026, September 18). Partnering with Accenture on Embedded Evaluation . https://www.anthropic.com/news/accenture-embedded-evaluation

• Singer, G. (2026, March 10). Emerging Technology and Engineering Challenges of Artificial Intelligence Deep scheming: New requirements for Containing Artificial Intelligence Misalignment. IEEE Micro. https://ieeexplore.ieee.org/document/11429121

• Greenblatt, R. et al. (2024, December 20). Alignment Faking in Large Language Models . arXiv. https://arxiv.org/pdf/2412.14093

• Singer, G. (2025, March 3). The Urgent Need for Intrinsic Alignment Technologies for Responsible Agentic AI. Towards Data Science. https://towardsdatascience.com/the-urgent-need-for-intrinsic-alignment-technologies-for-responsible-agentic-ai/

• Luo, X. et al. (2026, September 22). Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models. arXiv. https://arxiv.org/pdf/2609.26637