What Is Harness Engineering? A Hands-On Introduction

What Is Harness Engineering? A Hands-On Introduction | Towards Data Science

Agentic AI

What Is Harness Engineering? A Hands-On Introduction

Learn three essential primitives of harness engineering to future-proof your career

Sam Black

October 29, 2026

15 min read

Prefer us on Google

Share

Photo by DiEGO MüLLER on Unsplash

Let me start with a strong opinion.

If you're a software engineer or a data scientist, nothing you can learn today will do more for your career than learning to design agent harnesses.

It took me a little while to actually hit, but when I found out researchers at top firms hadn't written code in the last 8 months, it struck me.

We aren't going to be writing code anymore.

Let's clearly define what a "harness" is in the context of AI

There's a lot of confusion, buzzwords and conceptual conflation in the current state. Thus, we need a collective conceptual agreement on what a harness actually is.

You may have seen this equation before:

Agent

=

LLM

+

Memory

Tools

Planning

Action

\text{Agent} = \text{LLM} + \text{Memory} + \text{Tools} + \text{Planning} + \text{Action}

The LLM is the base. Stateless, unchanging, perhaps swappable.

The rest is the "harness". I list Memory, Tools, Planning and Action, but the universe of "extras" to orchestrate an agent's behavior can be much larger.

⏟

Harness

\text{Agent} = \text{LLM} + \underbrace{\text{Memory} + \text{Tools} + \text{Planning} + \text{Action}}_{\text{Harness}}

​

Thus, the world of agents is simply:

\text{Agent} = \text{LLM} + \text{Harness}

The important thing to understand is, the harness is everything we build around that model to turn its capabilities into an operational agent.

It's important to realize that Codex, Claude Code, among others, are just harnesses

Here, the LLM powering these tools is a second class citizen. Codex and Claude Code are agentic coding agents built around language models. They combine model inference with a collection of mechanisms (file editing, repls, git, etc.) that make autonomous software development possible.

The power lies in the cleverness of the harness.

This distinction changes where we should focus our engineering effort.

Suppose you want an agent to refactor a large codebase. You could spend weeks attempting to fine-tune a model that is marginally better at generating Python.

Alternatively, you could improve the harness so that the agent can map the repository, identify dependencies, construct a test plan, make incremental changes, run targeted tests, diagnose failures, and verify that unrelated behavior remains unchanged.

The second approach typically produces a much larger improvement in the overall system without needing to upgrade the underlying LLM.

Of course, model capability still matters. A harness cannot give a weak model the reasoning ability it lacks. But the performance of your agent isn't determined by model capability alone. It depends on how effectively that capability is applied to the task.

Which brings me to the central thesis: if you're a software engineer or data scientist, understanding harness engineering is no longer optional. It's essential to remaining relevant.

As your role shifts from writing code to producing agents who write code , designing the harness becomes the paramount skill. You need to create agents which can operate autonomously on problems which (potentially) span many complex domains. Said otherwise, you are no longer an individual contributor, even if you have no direct reports. You are a manager, a manager of agents.

You can't rely on Claude Code or Codex to solve all of your problems

These are good general-purpose harnesses. Very useful, but will fail spectacularly as soon as you approach a problem that is very complex or outside of the general purpose domain. If you rely on this approach, not only will you be spending your time on drift and slop cleaning, you'll also be making yourself less relevant.

A sad, but true fact: engineers who can only build systems using Claude Code or Codex are becoming commodities. You cannot remain competitive in the market if your only skill is "I know how to use Claude Code." Frankly, if this is your strategy for SWE in the AI era, you're a target for layoffs.

Your value lies in designing the harnesses that turn LLMs into systems that simply work.

So, how does one become a harness engineer?

This is not your "build an agent in 50 lines" post. You've probably seen the tutorial.

Call the model, check whether it asked for a tool, run the tool, append the result, loop until it says it's done. It's a great way to understand the mechanics, and it works in a demo. This is agent 101, the simplest loop that makes an LLM an agent.

python

while True:
response = model(messages)
if response.is_final:
break
result = run_tool(response.tool_call)
messages.append(response.tool_call)
messages.append(result)

However, this loop will not solve complex problems. Complex problems need memory, tools, context management techniques. The full harness.

This is about understanding the primitives underlying harnesses that actually work, so that you can design your own.

You've been promoted. You are no longer a senior software engineer. You are the core systems architect.

This is the key mindshift you need to make.

If you were previously comfortable writing software in the traditional manner: writing a spec, designing and implementing functions and unit tests, that job is no longer available . This is now the realm of AI agents.

This is good news. You have been automatically promoted to a systems architect.

Your role is now designing smart frameworks that allow agents to be successful in day-to-day operations. You're not writing the logic that solves the problem. You're designing systems that enable AI to solve problems.

These are fundamentally different activities. You need to think about your team of agents like junior software engineers. Where do you expect blockers? What tools do they need? How can you ensure that your feedback and design vision gets implemented correctly and reliably without you needing to micromanage all the details?

👉 I've broken this introduction into "primitives": engineering concepts that combine to produce a reliable agent harness. These primitives should be thought of like building blocks. Every harness should include them (in varying compositions)

Primitive #1. Workflows

Before any actual engineering occurs, you need to design a workflow . This is otherwise known as " Loop Engineering ."

A workflow for an agent is nearly the same as it is for a human. It is the sequence of stages, decision points and feedback loops that an agent follows in order to solve a task. Designing these workflows is where your domain expertise comes in.

For example, the quintessential data science workflow, as performed by a human. A stakeholder asks you to build a prediction model.

• Define the problem : What are we predicting, and what counts as success?

• Explore the data : Understand the data, identify issues, and find useful features.

• Build the model: Engineer features and train an initial model.

• Evaluate : Test performance on unseen data using relevant metrics.

• Iterate : Analyze errors, improve features, try different models, and repeat until performance is satisfactory.

• Deliver : Document the results and deploy the model.

This is the same principle behind successful engineering.

Break complex work into stages with verifiable outputs that can be refined with feedback.

Now, instead of a human executing each task, we design workflows in a loop such that they can be completed by an agent.

Some successful workflow design patterns:

• ReAct

• Plan-then-Execute

• Reflexion

• Sub-agents

ReAct

In ReAct: Synergizing Reasoning and Acting in Language Models , ICLR 2023, Yao et al. introduced ReAct (Reasoning + Acting), which is a conceptually simple pattern that interleaves the model's internal reasoning with concrete actions in an environment.

The structure alternates between three elements:

• Thought : the model reasons about the current state, what it knows, and what it should do next

• Action : the model invokes a tool or takes a step in the environment

• Observation : the result of that action is fed back to the model

For harness engineers , ReAct is the foundation. Nearly every production agent harness implements some version of this loop. The ReAct loop is the one referenced above. It's simplistically elegant.

However, it is limited by its linear nature. The agent follows a single thread of reasoning. When it makes a wrong turn, it often continues down the wrong path rather than backtracking. It doesn't have any memory, and it's context degrades quickly, using ReAct alone may not lead to success on complex problems.

Plan-then-Execute

In Plan-and-Solve Prompting (ACL 2023), Wang et al. investigated prompting models to formulate a plan before executing it. This is what every successful coding agent does in practice and can be used in multiple contexts outside of software engineering.

The pattern:

• Planning phase: Gather information, explore the codebase, read docs, identify all locations that need changes, produce a structured plan

• Review checkpoint: The plan is verified (by a human, by a validation step, or by a self-review)

• Execution phase: Follow the plan step by step, with the harness providing context about the current step and tracking progress

Every software engineer knows the difference between jumping into code versus writing a design doc first. Plan-then-Execute is that intuition, formalized as a harness.

If ReAct is the foundation for simple workflows, Plan-then-Execute is the foundation for complex workflows.

Reflexion

Shinn et al.'s Reflexion ( Reflexion: Language Agents with Verbal Reinforcement Learning , NeurIPS 2023) is the pattern of iterative self-improvement through reflection on failure . The agent attempts a task, evaluates the result, generates verbal feedback about what went wrong, stores that feedback in memory, and tries again with the benefit of that reflection.

This is directly what happens in every coding agent when tests fail. The agent writes code, runs tests, sees failures, reflects on what went wrong, and revises. Reflexion formalized this and showed an 11-point improvement on HumanEval (80% → 91%) with no model change.

The model uses a metacognitive strategy to understand its own failure, then uses that as context to improve the final result.

Sub-agents

The idea is simple: decompose a large task into smaller subtasks, and delegate each subtask to a specialized agent with its own tools and context window. A supervisor manages the delegation, collects results, and synthesizes them into a final output.

The nice thing about this pattern is that each sub-agent can use an elegant workflow as described above. The core limitations here are the strength of the delegating agent and the robustness of your workflow.

The key design decisions when working with sub-agents are:

• Scope boundaries : What does each sub-agent own? Clear ownership prevents agents from stepping on each other's work.

• Interface contracts : What information does the coordinator pass to each sub-agent, and what does it expect back? The more structured these contracts, the more reliable the system.

• Failure isolation : If a sub-agent fails or produces garbage, the coordinator needs to detect this and either retry, reassign, or escalate, without contaminating the work of other sub-agents.

• Context independence : Each sub-agent should be able to operate with only the information it is given. If a sub-agent needs to read the entire 200-file repository to do its job, you've scoped it wrong.

These workflow patterns are essential, regardless of whether you're building harnesses for software engineering, data science or some other domain.

It's also important to note that, these are just foundational examples. They are by no means requisite. For example, you might design a hybrid: A sub-agent delegate system where each sub-agent uses reflexion + a human-in-the-loop step. These patterns are established enough to be used like building blocks.

⚠️ Don't reinvent the wheel. Improve or modify these foundations as you see fit.

Primitive #2: Architecture

Evolving a harness from something simple, like autoresearch, to something more complex, like Codex requires foresight and planning. There needs to be clear architectural and structural boundaries that organize the harness such that the agent can operate cleanly and even modify its own harness if necessary without creating massive amounts of technical debt that needs a human to clean.

If you've built production software in a team, you already have intuitions here.

First, design the scaffolding of your harness. This organizes the tools, memory, prompts, etc. into something coherent and legible.

text

your-harness/
├── config/
│   ├── models.yaml              # Provider configs, API keys, rate limits
├── model/
│   ├── interface.py             # Unified model calling API
│   ├── context.py               # Context window management, summarization
│   └── providers/
│       ├── openai.py
│       └── anthropic.py
├── tools/
│   ├── registry.py              # Tool registration, lookup, schema validation
│   ├── builtin/                 # Your tools (each file = one tool + its schema)
│   └── custom/                  # Agent-generated tools land here
├── state/
│   ├── memory.py                # Working memory, checkpointing, rollback
│   └── plan.py                  # Plan tracking, step completion status
├── orchestration/
│   ├── loop.py                  # Core agent loop
│   ├── router.py                # Next-action decisions: tool, sub-agent, escalate
│   └── subagents.py             # Sub-agent spawning and coordination
├── evaluation/
│   ├── linters.py               # Custom linters
│   └── validators.py            # Output validation (schema, tests, review)
├── observability/
│   ├── tracer.py                # Structured trace logging
│   └── metrics.py               # Cost, latency, success rate tracking
├── prompts/
│   ├── system.md.jinja          # Base system prompt template
│   └── stages/                  # One template per workflow stage
├── evals/                       # Benchmark tasks for regression testing
├── main.py
└── README.md

Every project is different, but starting with a clear mental model for how the agent is structured is necessary. Remember, it's likely to evolve, so keep things as clear as possible without overlaps which can confuse agents who attempt to modify the harness.

Compare this model to something like Hermes agent, which has likely suffered from considerable AI drift. Its root AGENTS.md has 14 different sub AGENTS.md. The /agents directory, which contains most of the agents' actual logic has 395 separate files and 145k lines of code. This doesn't even include the skills or prompts. I'm not saying it's unnecessary for Hermes to need this degree of complexity, but it's most likely safe to say that some AI-generated drift has occurred here.

Not enforcing a clear boundary on the harness itself and allowing the agent to contribute to its own harness is a recipe for technical debt. Although the complexity of the project and overlapping modules aren't clear evidence of AI drift, its still my best guess.

A well-scaffolded harness separates at least the following concerns:

• The model interface layer : The boundary between your harness and the LLM. It handles prompt construction, token counting, response parsing, retry logic, and model-specific quirks. If you swap from Claude to GPT-4 to Gemini, only this layer should change.

• The tool/skills layer: Tool definitions, input validation, execution. Each tool should be a self-contained unit with a clear schema and document.

• The state layer : The agent's working memory, task progress, accumulated context, plan state, and any artifacts it has produced.

• The orchestration layer: This is where your workflow logic lives. It decides what happens next: which phase the agent is in, whether to invoke a sub-agent, whether to checkpoint, whether to escalate to a human.

• The evaluation layer: Mechanisms that assess whether the agent's outputs meet quality thresholds. This can be anything from running unit tests to having a second model review the output to checking structural constraints on the result. This layer gates transitions between workflow stages.

The success criterion is that a human or agent can reason about each one independently, test each one independently, and modify each one without worrying about cascading side effects.

This scaffolding serves a dual purpose. It makes the system maintainable for you. And it makes the system legible to the agent operating within it. An agent that can read its own tool definitions, understand its own state schema, and see the structure of its own workflow is an agent that can be trusted with more autonomy.

Self-Modifying Harnesses

The most powerful harness architectures allow the agent to extend its own capabilities. In a practical, engineering sense: the agent encounters a recurring problem, writes a new tool to solve it, registers that tool in its tool layer, and uses it going forward.

This is already happening. Coding agents routinely write helper scripts, create utility functions, and build small tools mid-task to make subsequent steps easier. The question is whether your architecture treats this as an accident or a first-class capability.

To support safe self-modification, you need:

• A tool registry that is modular : If tools are defined as configuration (schemas, descriptions, entry points) rather than hardcoded into the orchestration logic, the agent can add new tools by simply writing a new configuration entry and the corresponding implementation. It doesn't need to touch the core harness.

• Sandboxed execution : Any tool the agent creates must run in a sandbox. It should not have access to the harness internals, your credentials, or anything outside its designated scope. The harness trusts its built-in tools because you wrote them. Agent-authored tools earn no such trust by default.

• Validation before registration : Before a new tool becomes available, it should pass basic checks. Does the schema parse? Does a smoke test succeed? Does it conflict with existing tools? Use auto linters.

📌The scaffolding is what makes this safe. Without it, self-modification is a recipe for the kind of cascading, incomprehensible technical debt that grinds systems to a halt.

Primitive #3. Observability

Agent systems give you none of the deterministic safeguards that traditional software systems (typically) do by default with unit tests. The model is a black box. Its "reasoning" is a stream of tokens. Its decisions are probabilistic: the interaction between the model and the harness produces emergent behavior that can be extremely difficult to reason about after the fact.

Observability is how you make agent systems trustworthy. With a good observability stack, you can diagnose failures and iteratively improve.

Trace Everything

The absolute minimum: log every model call and every tool invocation with full inputs and outputs. Every single one. This is your equivalent of a stack trace.

A single agent task might involve dozens of model calls and tool invocations. When something goes wrong (and it will), you need to be able to reconstruct exactly what happened:

• What was in the prompt, reasoning and response at each step?

• What tool was called, with what arguments and what did it return?

• How did the harness interpret that result?

• What decision did your workflow orchestration make next, and why?

Build it in from the start. Every model call should pass through a logging wrapper. Every tool invocation should be recorded. Every state transition should be annotated.

Use structured formats (JSON, spans with parent-child relationships) that you can query, filter, and visualize. You want to be able to answer questions like: "Show me every task in the last week where the agent retried a tool call more than 3 times" or "What was the average context window utilization at the point where tasks started failing?"

An example schema:

json

{
"schema_version": "1.0",
"event_id": "evt_000042",
"trace_id": "trace_00121",
"task_id": "task_001",
"session_id": "session_007",
"parent_event_id": "evt_000041",
"sequence": 42,
"timestamp": "2026-10-23T16:00:01.234Z",
"event_type": "model.call",
"component": "agent_runtime",
"status": "success",
"duration_ms": 842,
"input": {},
"output": {},
"metadata": {
"retry": 0,
"worker_id": "worker_01",
"tags": ["demo"]
},
"error": null
}

Human Review Loops

Finally, observability includes the ability for humans to inspect and intervene in agent behavior. This is not the same as a human-in-the-loop workflow step (though that's valuable too). This is the ability for an engineer to:

• Watch an agent work in real time, seeing its reasoning and actions as they happen

• Pause an agent mid-task to inspect state and context

• Replay a completed task step-by-step to understand behavior

• Flag specific agent behaviors for review and annotation

These capabilities are what transform a harness into a system you can actually improve with confidence. Pinpoint where an agent is failing and design the correct behavior.

Wrapping up

The trajectory is clear. Models will continue to get more capable. But the gap between raw model capability and a system that reliably solves real problems will remain. That gap is the harness. That gap is now your job.

The engineers who thrive in the next decade won't be the ones who write the most code, or even the ones who prompt most cleverly. They'll be the ones who can design, build, and operate the systems that turn model intelligence into reliable outcomes. Workflow design. Architecture. Observability. These are the primitives. Master them, and you'll be building the infrastructure that everyone else depends on.

Thanks for reading!

References

[1] Lopopolo, R. (2026, February 11). Harness engineering: Leveraging Codex in an agent-first world . OpenAI. https://openai.com/index/harness-engineering/

[2] Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36 , 8634–8652. https://doi.org/10.48550/arxiv.2303.11366

[3] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023). Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2609–2634. https://doi.org/10.18653/v1/2023.acl-long.147

[4] Weng, L. (2026, July 4). Harness engineering for self-improvement . Lil'Log. https://lilianweng.github.io/posts/2026-07-04-harness/

[5] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR) . https://doi.org/10.48550/arxiv.2210.03629