Identifying Token Costs Hiding in Your Agentic Loop

Identifying Token Costs Hiding in Your Agentic Loop - MachineLearningMastery.com

Identifying Token Costs Hiding in Your Agentic Loop

By

Vinod Chugani

on

August 7, 2026

in

Artificial Intelligence

0

Share

Post

In this article, you will learn how token costs silently compound in agentic AI loops, and what architectural patterns you can use to control them before they escalate.

Topics we will cover include:

• Why token costs compound non-linearly in multi-step agentic workflows, and how the distinction between state and context is central to controlling them.

• Five distinct failure modes — from O(N²) context accumulation to static system prompt duplication — that account for the bulk of runaway token spend in production deployments.

• Practical mitigations for each trap, including context compaction, circuit breakers, payload filtering, dynamic model routing, and runtime prompt injection.

Time is money, and in agentic systems, tokens are both.

THe Core Issue

Building a single-turn LLM wrapper is a weekend project. Keeping an autonomous agent from silently bankrupting your infrastructure over a six-month deployment is a different problem entirely.

Here’s the core issue: every time an LLM processes text, it charges you in tokens , the small chunks of text (roughly three-quarters of a word each) that models use to read and write. Think of tokens as the metered units on your cloud bill. The more tokens you send per API call, the more you pay. Simple enough for a chatbot. But in an agentic loop — where an AI autonomously calls tools, reads results, and plans its next move across dozens of steps — token costs don’t grow linearly. They compound. A naive setup that dumps every tool output into an ever-growing message array can turn a \$0.05 automation task into a \$5.00 infinite loop without triggering a single error.

The fix starts with a clean mental distinction: State , which is the minimum facts needed to move the task forward, versus context , the full, verbose transcript of everything that’s happened so far. Most agentic frameworks confuse the two by default, and if you’re evaluating which frameworks are worth your time before architecting around them, this breakdown of the leading AI agent frameworks in 2025 is worth reading first. The five cost traps below are what that state/context confusion looks like in production.

Each trap below represents a distinct failure mode, some of which are deceptively simple, while others are surprisingly subtle. Taken together, they account for the bulk of runaway token spend in real deployments.

1. The O(N²) Context Accumulation Tax

The Concept: In an agentic loop, passing the full conversation history to every model call means you pay for the same historical tokens repeatedly, not just once.

How It Works: Most orchestration frameworks default to appending every user , assistant , and tool message to a single growing array. By step 20 of a 20-step workflow, the model re-reads everything from steps 1 through 19. The fix is context compaction : collapsing previous turns into a dense rolling summary, or using KV-cache prompt caching to freeze the prefix state and only pay for the delta — a direct consequence of how attention mechanisms scale with sequence length.

Worth Noting: Compress too aggressively and you get “context amnesia.” The agent drops a critical parameter it retrieved in step 2, hallucinates a replacement in step 8, and cascades into a chain of failed downstream tool calls.

When to Use It: Apply context compaction to any multi-step workflow expected to exceed five turns or interact with high-latency, data-heavy external APIs.

2. Unbounded Retry Loops on Stale State

Context bloat isn’t just an accumulation problem. It gets actively worse when things go wrong.

The Concept: When a tool call fails, the agent tries to self-correct but drags the full bloated context of the failure along for every retry, compounding costs with each attempt.

How It Works: A standard ReAct (Reasoning and Acting) loop catches an exception — say, a 400 Bad Request — and appends the error trace to the context before asking the model to fix it. If the agent gets stuck, each retry sends all previous failures too. The solution is a circuit breaker at the orchestrator level: strip failed trajectories from the state before presenting the error back to the model, or halt execution entirely after a threshold.

Worth Noting: Stripping the failure history completely means the agent will likely repeat the exact same invalid tool call. You need to extract and inject a deterministic “failure heuristic” (e.g. “Tool X failed because parameter Y was missing”) rather than the raw stack trace.

When to Use It: Enforce circuit breakers and trajectory pruning on all non-deterministic external API calls where the model dynamically generates the payload.

3. Unfiltered Tool Payload Bloat

With retry loops under control, the next place to look is what gets fed into the context in the first place — specifically, the raw output from your tools.

The Concept: Feeding raw, unparsed API responses directly into the agent’s context wastes tokens on structural boilerplate and fields the agent will never use.

How It Works: An agent queries a database or third-party API and gets back a massive JSON payload. Instead of dumping that raw JSON into the prompt, route it through a deterministic extraction layer ( jq , a regex filter, or a dedicated parser) that strips metadata, null fields, and boilerplate. What goes into the context should be only the schema-validated key-value pairs the agent actually needs to move forward.

Worth Noting: If the extraction layer quietly drops a field the agent needs downstream, it will silently hallucinate a plausible value to fill the gap — and that value goes straight into your database writes.

When to Use It: Deploy payload filtering middleware whenever an agent integrates with legacy systems, verbose REST APIs, or unstructured web scraping tools.

4. Monolithic Model Routing

Once your context is lean and your payloads are filtered, there’s still a cost lever most engineers ignore: which model is doing the work.

The Concept: Defaulting to your most capable (and expensive) model for every step in a workflow — including trivial tasks like formatting a JSON object or classifying an intent.

How It Works: An agentic workflow is really a directed graph of heterogeneous tasks. Complex semantic reasoning and planning warrant a heavyweight model. But for nodes handling intent classification, JSON formatting, or schema validation, the orchestrator can dynamically route to a smaller, cheaper model (e.g. Llama 3 8B or GPT-4o-mini) at a fraction of the token cost.

Worth Noting: Routing adds orchestration overhead. If your system has to load a different model into VRAM or open a new provider connection at every step, the latency hit can wipe out the savings.

When to Use It: Dynamic model routing pays off in high-throughput, multi-agent systems where the workflow graph contains clearly isolated nodes for deterministic data transformation.

5. Static Context Duplication

The last trap lives at the very top of every API call, in the system prompt itself.

The Concept: Injecting one massive system prompt covering every tool definition and edge case into every single API call, even when most of it is irrelevant to the current step.

How It Works: Rather than loading a 5,000-token system prompt defining 20 tools, build your prompts dynamically using the techniques covered here . The orchestrator keeps a vector index or lightweight rules engine of available tools and constraints. At runtime, it injects only the tool definitions and behavioral guidelines the current step actually needs — nothing more.

Worth Noting: Dynamic context injection opens a prompt injection vulnerability if the lookup query is influenced by untrusted user input. A maliciously crafted query could cause the orchestrator to retrieve and execute a tampered tool definition.

When to Use It: Switch to dynamic prompt construction when your agent’s tool count exceeds a dozen, or when you’re running multi-tenant systems with distinct role-based access controls.

Managing Token Costs in Production

These five traps share a common root cause: treating context as unlimited. Once you start managing it deliberately — compacting history, pruning failures, filtering payloads, routing by task complexity, and injecting only what each step needs — the cost profile of your agentic system changes substantially.

The token costs hiding in your agentic loop

But cutting your runtime token burn is just the first problem. By day 100 in production, you’ll be dealing with compounding infrastructure costs from state management. Storing massive, uncompressed agent trajectories for observability or crash recovery will bloat your storage and degrade query latency fast. Implement aggressive TTLs on session states and cold-storage archiving for long-term audit logs, so your operational database only holds active, high-priority state.

Tokens are the compute currency of agentic systems. Treating them as a free resource is a reliable way to fail in production. Don’t wait for model providers to lower their API pricing. Architect your orchestration layer to treat context as a constrained, volatile resource from day one.

More On This Topic

• Implementing Prompt Compression to Reduce Agentic Loop Costs

• How AI Cuts Costs and Adds Value for Data Science…

• Build an Inference Cache to Save Costs in…

• An Introduction to Loop Engineering

• Building a 'Human-in-the-Loop' Approval Gate for…

• Creating a Training Loop for PyTorch Models

/.entry