# What is an agent harness? Meaning, examples and a four-harness test

An agent harness is the software that turns a model into an agent: its loop, tools, context and permissions. We tested Claude Code, Codex, OpenCode and Pi.

- Published: 2026-11-22
- Canonical: https://replicas.dev/resources/agent-harness

An agent harness is the software around a language model that turns it into an agent. It runs the loop that sends the model a task and a list of tools, carries out the tool calls the model asks for, decides what goes into the model's context, and enforces what the agent is allowed to do. Claude Code, Codex, OpenCode and Pi are agent harnesses for coding.

The model on its own only reads text and writes text back. The harness is what lets it edit files and run commands. LangChain sums it up as "Agent = Model + Harness" ([LangChain](https://www.langchain.com/blog/the-anatomy-of-an-agent-harness)). Other names for it are agentic harness, AI agent harness and agent scaffolding ([Wikipedia](https://en.wikipedia.org/wiki/Agent_harness)). "LLM harness" is used for it too, but also for evaluation tools that run benchmarks against a model, such as EleutherAI's [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).

This guide explains what a harness contains and how it differs from a model, a framework and an agent. Then it reports tests we ran on November 22, 2026: we gave the same coding task to four harnesses, three times for each harness and model, and held the model constant to see what the harness changes.

## What an agent harness does

Most definitions list the same parts ([Wikipedia](https://en.wikipedia.org/wiki/Agent_harness), [LangChain](https://www.langchain.com/blog/the-anatomy-of-an-agent-harness), [VS Code](https://code.visualstudio.com/docs/agents/concepts/agent-harnesses)). The last column shows what each part looked like in our test.

| Part | What it does | In our test |
|---|---|---|
| Agent loop | Sends the conversation to the model, runs the tool calls it returns, adds the results and sends it again, until the model stops | 4 to 11 model requests for the same task |
| Tools | The actions the model can ask for, such as reading and editing files, running shell commands, searching the web or starting a subagent | 4 tools in Pi, 11 in Claude Code plus 13 more it loads on request |
| Instructions | A fixed system prompt, details about the machine, and project files such as `AGENTS.md` | 1,343 to 17,698 tokens before the model had read any code |
| Context management | What stays in the context window as a session grows: prompt caching, trimming old output, and summarizing old turns (compaction) | Later sessions read part or all of the fixed prompt from the provider's cache |
| Permissions and sandbox | Which actions run without asking, which need approval, and what the process can reach | With default settings, two of the four harnesses changed nothing |
| Model settings | Which model to call, how much reasoning effort to ask for, and the output limit | Claude Sonnet 5.5 ran at low, medium or high effort depending on the harness |

## Agent harness vs model, framework and agent

These four terms are often mixed up. A model can run in many harnesses, and many harnesses can run more than one model.

| Term | What it is | Examples |
|---|---|---|
| Model | The language model. It has no memory between requests and can't act on its own | Claude Sonnet 5.5, GPT-6.2-Sol |
| Agent harness | The program that runs a model in a loop with tools, instructions and rules | Claude Code, Codex, OpenCode, Pi |
| Agent framework | A library for building your own harness | Microsoft Agent Framework ([Microsoft Learn](https://learn.microsoft.com/en-us/agent-framework/concepts/harness)) |
| Agent | A harness running a model on a task | Claude Code with Claude Sonnet 5.5 fixing a bug |

So Claude Code is a harness, and a Claude Code session working on a task is an agent. VS Code's docs use the word the same way: its chat can run the Copilot, Claude or Codex harness, and the harness you pick decides the tools, the models on offer and the approval modes ([VS Code](https://code.visualstudio.com/docs/agents/concepts/agent-harnesses)).


## What is harness engineering?

Harness engineering is the work of building the parts of a coding agent's harness that your team controls, so the agent does reliable work in your codebase. OpenAI introduced the term in February 2026 in a post about building a product with Codex and "0 lines of manually-written code". It describes the engineer's job as to "design environments, specify intent, and build feedback loops" ([OpenAI](https://openai.com/index/harness-engineering/)).

Birgitta Böckeler explains how it relates to the harness itself ([martinfowler.com](https://martinfowler.com/articles/harness-engineering.html)). Part of a coding agent's harness is already built in, such as its system prompt and tools, and agents also let users build an "outer harness" for their own codebase. She calls that work "a specific form of context engineering", and splits it into two kinds of control:

| Kind of control | What it does | Examples |
|---|---|---|
| Guides | Steer the agent before it acts | An `AGENTS.md` file with your conventions, skills |
| Sensors | Check the result after it acts, so the agent can correct itself | Linters, type checkers, structural tests in a pre-commit hook, AI code review |

OpenAI's post gives concrete examples of both: an `AGENTS.md` kept short as "the table of contents" for a `docs/` folder in the repository, and architecture rules "enforced mechanically via custom linters" and structural tests.

Our test used the smallest possible guide: a two-line `AGENTS.md` asking agents to log every fix in `CHANGES.md`. All four harnesses loaded it, and every run followed it, even though the prompt never mentioned the file. The rest of this guide is about the built-in part of the harness, which you choose rather than build.

## The same task through four harnesses

We used a real bug in [humanize](https://github.com/python-humanize/humanize), an MIT-licensed Python library: `ordinal(-1)` returns `-1th` instead of `-1st`. Each harness got the same prompt in a fresh copy of the repository, and the repository's `AGENTS.md` asked agents to record every fix in `CHANGES.md`. The prompt didn't mention that file, so a changelog line shows the harness had passed `AGENTS.md` to the model.

We ran Claude Code with Claude Sonnet 5.5 and Codex with GPT-6.2-Sol, and ran OpenCode and Pi with both models. That gives two groups where the model is the same and only the harness changes. [How we tested](#how-we-tested) has the details.

Median of three runs per harness on November 22, 2026, with cost at list prices.

| Model and harness | Time per run | Model requests | First request, in tokens | Cost per run |
| --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 in Claude Code | 20 s | 4 | 17,698 | $0.067 |
| Claude Sonnet 5.5 in OpenCode | 28 s | 6 | 10,779 | $0.062 |
| Claude Sonnet 5.5 in Pi | 20 s | 4 | 2,638 | $0.028 |
| GPT-6.2-Sol in Codex | 29 s | 6 | 11,756 | $0.062 |
| GPT-6.2-Sol in OpenCode | 37 s | 11 | 5,037 | $0.051 |
| GPT-6.2-Sol in Pi | 24 s | 7 | 1,343 | $0.030 |

The first request is everything the harness sent before the model had looked at any code: its system prompt, tool definitions, `AGENTS.md`, environment details and our prompt. Anthropic and OpenAI count tokens differently, so compare token counts within a model group, not across groups. Costs use each provider's list prices ([Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [OpenAI](https://developers.openai.com/api/docs/pricing)).

- **All 18 runs did the job.** Every run fixed the bug, added regression tests, passed the full test suite and added the changelog line that `AGENTS.md` asked for.
- **Cost changed more than the result.** With the same model, the most expensive harness cost about 2.4 times as much per run as the cheapest with Claude Sonnet 5.5, and about 2.1 times as much with GPT-6.2-Sol.
- **Pi was the cheapest with both models.** It sends the shortest prompt and only four tools, and it asked Claude Sonnet 5.5 for low effort. Claude Code asked for medium effort, and OpenCode set no effort, so the API default of high applied ([Anthropic](https://platform.claude.com/docs/en/build-with-claude/effort)).
- **This was one small task.** A bigger prompt or more tools can pay off on harder work, and this test can't show that.

### A second task, with reasoning effort set to high

The harnesses asked for different reasoning effort in the first test, so we ran a second task with each harness set to high effort: `--effort high` in Claude Code, `model_reasoning_effort=high` in Codex, `--variant high` in OpenCode and `--thinking high` in Pi. The task was a small Node.js project we wrote for the test. One test failed because tax was charged before the discount, and the agent had to fix that, add a `BULK10` discount code with tests, and document it. We checked every run against seven hidden tests the agents never saw. This time the Anthropic model was Claude Opus 5.5.

| Model and harness | Time per run | Model requests | First request, in tokens | Cost per run |
|---|---:|---:|---:|---:|
| Claude Opus 5.5 in Claude Code | 20 s | 4 | 17,559 | $0.160 |
| Claude Opus 5.5 in OpenCode | 23 s | 5 | 10,639 | $0.128 |
| Claude Opus 5.5 in Pi | 19 s | 5 | 2,474 | $0.071 |
| GPT-6.2-Sol in Codex | 25 s | 7 | 14,919 | $0.078 |
| GPT-6.2-Sol in OpenCode | 36 s | 12 | 4,959 | $0.054 |
| GPT-6.2-Sol in Pi | 21 s | 6 | 1,252 | $0.026 |

All 18 runs passed the hidden tests and the project's original tests. Pi was again the cheapest with both models, and the gap between the cheapest and most expensive harness was about 2.3 times on Claude Opus 5.5 and 3 times on GPT-6.2-Sol. As in the first test, we priced each run's first request as if the cache were empty. Claude Code's runs actually read about 12,000 tokens from a cache that earlier sessions on the same API account had warmed, which brought its median to $0.103. With that cache hit, it cost less than OpenCode.

## What changed between the harnesses

### Tools

| Harness | Tools in the first request |
|---|---|
| Claude Code | 11: `Agent`, `Bash`, `Edit`, `ListAgents`, `Read`, `ReportFindings`, `ScheduleWakeup`, `Skill`, `ToolSearch`, `Workflow`, `Write`. It also listed 13 more by name, such as `WebFetch` and `WebSearch`, which load only when the model asks for them through `ToolSearch` |
| Codex | 9: `exec_command`, `write_stdin`, `request_user_input`, `view_image`, `multi_agent_v1`, `get_goal`, `create_goal`, `update_goal`, `web_search`. It edits files with `apply_patch` run through the shell. In the second task, with high effort and its `workspace-write` sandbox, Codex instead gave the model an `exec` tool that runs JavaScript, which called `exec_command` and `apply_patch`, sometimes several at once |
| OpenCode | 10 with Claude Sonnet 5.5: `bash`, `edit`, `glob`, `grep`, `read`, `skill`, `task`, `todowrite`, `webfetch`, `write`. With GPT-6.2-Sol, `apply_patch` replaced `edit` and `write` |
| Pi | 4: `read`, `bash`, `edit`, `write` |

These lists come from the requests each harness sent in headless mode with no MCP servers or skills installed. Interactive sessions and extra servers change them. For a tool-by-tool comparison of Claude Code and OpenCode, see [OpenCode vs Claude Code](/resources/opencode-vs-claude-code).

### Context and instructions

| Harness | Reads `AGENTS.md` | Read from the previous session's cache in our runs |
|---|---|---|
| Claude Code | Yes, when the repository has no `CLAUDE.md` ([Claude Code docs](https://code.claude.com/docs/en/memory)) | 11,348 tokens of its prompt |
| Codex | Yes ([Codex docs](https://learn.chatgpt.com/docs/agent-configuration/agents-md)) | None |
| OpenCode | Yes, and reads `CLAUDE.md` where there is no `AGENTS.md` ([OpenCode docs](https://opencode.ai/docs/rules)) | With Claude Sonnet 5.5, all 10,754 tokens when the next session ran in the same folder, and none in a new folder. With GPT-6.2-Sol, none |
| Pi | Yes, and `CLAUDE.md` files too ([Pi docs](https://docs.pi.dev/latest/security)) | 1,310 tokens with Claude Sonnet 5.5. None with GPT-6.2-Sol |

All four harnesses put the `AGENTS.md` text into their first model request, and every run followed it.

Caching is why a large fixed prompt costs less than its size suggests. When a later request starts with the same text, the provider reads it from its cache at a twentieth of the normal input price ([Anthropic](https://platform.claude.com/docs/en/about-claude/pricing), [OpenAI](https://developers.openai.com/api/docs/pricing)). Within one session, every harness got cache hits from the second request on. Between sessions, it depends on whether the start of the prompt changes. OpenCode puts the working folder into the cached part of its prompt, so a new folder means a new cache entry. With GPT-6.2-Sol, no harness got a cache hit on a session's first request, and all three sent a different `prompt_cache_key` for each session.

None of our runs came close to a context limit. The largest request was about 22,000 tokens, so this test says nothing about how each harness summarizes a long session.

### Permissions

We also ran each harness once with no permission flags, as a script or CI job would.

| Harness | Command | What happened with default settings | Flag we used for the main runs |
|---|---|---|---|
| Claude Code | `claude -p` | Started in Manual mode. Claude Code denied the file edits, changed nothing and described the fix it would have made ([Claude Code docs](https://code.claude.com/docs/en/permission-modes)) | `--dangerously-skip-permissions` |
| Codex | `codex exec` | Ran in a read-only sandbox. Codex tested the fix in memory, printed a diff and changed nothing ([Codex docs](https://learn.chatgpt.com/docs/agent-approvals-security)) | `--dangerously-bypass-approvals-and-sandbox` |
| OpenCode | `opencode run` | Made the fix. OpenCode's docs describe a default policy that only asks before reading `.env` files ([OpenCode docs](https://opencode.ai/docs/permissions)), but in the second task's probe `opencode run` 1.19.4 still rejected a write outside the project | `--auto` |
| Pi | `pi -p` | Made the fix. Pi has no approval step for tool calls and no built-in sandbox ([Pi docs](https://docs.pi.dev/latest/security)) | None needed |

In the second task we used narrower settings and asked each harness to write a file outside the project and fetch a web page. Codex's `workspace-write` sandbox blocked both, because it turns off network access ([Codex docs](https://learn.chatgpt.com/docs/agent-approvals-security)). Claude Code in `acceptEdits` mode blocked the outside write, and fetched the page only because we had allowed `Bash`. OpenCode's `run` rejected the outside write and fetched the page. Pi did both.

The flags in the last column turn off the checks, so only use them where the agent can't reach anything you'd mind losing. [How to run Claude Code in a sandbox](/resources/claude-code-sandbox) compares the safer ways to do that.

### Model settings

The model was the same within each group, but the harnesses called it differently.

| Harness | Reasoning effort it asked for | Output limit | Extra requests |
|---|---|---|---|
| Claude Code, with Claude Sonnet 5.5 | Medium | 128,000 tokens | None |
| OpenCode, with Claude Sonnet 5.5 | None set, so the API default of high | 32,000 tokens | One request to Claude Haiku 5.5 to title the session |
| Pi, with Claude Sonnet 5.5 | Low | 128,000 tokens | None |
| Codex, with GPT-6.2-Sol | None set | Not set | None |
| OpenCode, with GPT-6.2-Sol | None set | Not set | One request to GPT-5.4 nano to title the session |
| Pi, with GPT-6.2-Sol | Medium | 128,000 tokens | None |

Each harness lets you set the reasoning effort, for example with `--effort` in Claude Code, `--variant` in OpenCode and `--thinking` in Pi. In the first task we left every harness at its defaults, because that is what most people run.

## Does the harness change the result?

In our tests it didn't: all 36 runs passed. A larger study came to a similar conclusion. Yangze Liu and Zhongyi Han ran seven models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified and two other benchmarks ([arXiv](https://arxiv.org/abs/2610.04433)). No harness led across all models. Where they could measure it, the gap in pass rate was 2 to 5 points, and swapping the harness changed about as many task outcomes as rerunning the same one. Cost differences between harnesses ranged from under 10% to three or four times, depending on the model.

So when you choose a harness, the cost per task, the speed, the default permissions and how well it fits your tools matter as much as benchmark scores. These pages compare specific harnesses:

- [Claude Code alternatives](/resources/claude-code-alternatives), tested on the same four tasks
- [Codex alternatives](/resources/codex-alternatives)
- [OpenCode vs Claude Code](/resources/opencode-vs-claude-code), including token overhead
- [OpenCode vs Codex](/resources/opencode-vs-codex)
- [Pi vs OpenCode](/resources/pi-vs-opencode)
- [Codex vs Claude Code](/resources/codex-vs-claude-code)

## How we tested

An AI agent working in a Replicas workspace set up and ran these tests on November 22, 2026, and a person on our team reviewed this guide before it was published. Replicas is our product.

- **Machine.** One Replicas workspace, a Linux VM. We ran each harness's own command-line tool in headless mode, not through the Replicas chat, so Replicas' own instructions aren't in these numbers.
- **Versions.** Claude Code 2.1.330, Codex CLI 0.167.0, OpenCode 1.19.4 and Pi 1.6.0.
- **Models.** Claude Sonnet 5.5 with an Anthropic API key, and GPT-6.2-Sol with an OpenAI API key.
- **Repository.** humanize at commit `a950045`, plus a two-line `AGENTS.md` and an empty `CHANGES.md`. There was no customer code or data.
- **Setup.** Every run started from a fresh clone in a new folder and an empty home folder, so no user settings, skills or MCP servers were loaded. For the main runs we turned approvals off with each harness's flag. We ran one warm-up round, then three measured runs of each pairing.
- **Measurement.** A local proxy recorded every request each harness sent and the token counts the API returned. Time is wall-clock time from start to exit.
- **Checks.** We compared `ordinal()` against the correct suffix for every integer from -300 to 300, ran the full test suite, and checked `CHANGES.md` for the new line.
- **Cost.** Each run's cost uses the token counts the API returned and the list prices on the day. Because every run used a new folder, we priced each run's first request as if the cache were empty, so a harness wasn't penalized for having the folder in its cached prompt.
- **A mistake we fixed.** In our first Codex runs, Codex's home folder sat under `/tmp`, where Codex won't install its `apply_patch` helper, so it spent extra requests working around that. We moved the home folder and ran Codex again. The table shows the second set.

The prompt we gave every harness:

```text
humanize.ordinal() returns the wrong suffix for some negative numbers: ordinal(-1) gives "-1th" and ordinal(-2) gives "-2th". Fix it so a negative number gets the same suffix as its positive counterpart (-1st, -2nd, -3rd, -11th, -12th, -101st), add regression tests, and run the test suite. Don't commit.
```

The second task ran the same way: Claude Code 2.1.330, Codex CLI 0.167.0, OpenCode 1.19.4 and Pi 1.6.0, a fresh copy of the project and an empty home folder for every run, three runs per pairing, effort set to high, and costs from the token counts each harness reported. Its Codex runs kept the home folder under `/tmp`. Codex still applied its patches, but its numbers may differ from a normal install.

For the Cursor example, we started one Replicas workspace with Cursor's default model through the Replicas CLI (`replicas create -a cursor`), in an environment with no repositories, and gave it the first task's prompt plus a step to clone humanize. We ran it once and didn't measure its cost.

These are two small tasks run three times per pairing, not a benchmark. We didn't test long sessions, compaction, MCP servers, interactive mode, subscription sign-ins, or how each harness handles a task it can't finish.

## One harness, or several on one platform

Some coding agent products are a single harness built around their own agent. Devin's docs describe one agent, Devin, working in a Linux virtual machine "with your repositories cloned, tools installed, dependencies resolved" ([Devin docs](https://docs.devin.ai/onboard-devin/environment)). Cursor runs its own agent harness in its editor, its CLI and its Cloud Agents, which "run in isolated VMs in the cloud" with "a curated selection of models" ([Cursor docs](https://cursor.com/docs/cloud-agent)). In both products, the company that makes the harness also runs the machine it works in.

Other platforms run several harnesses in the same place. Replicas is one of them: Claude Code, Codex, Cursor, Gemini (through Google's Antigravity CLI), Muse Code, OpenCode and Pi each run as their vendor ships them, inside the same kind of cloud VM ([Replicas docs](https://docs.replicas.dev/admin/credentials)). Cursor is a good example of the difference. In Replicas, Cursor's own harness runs inside the workspace through the Cursor Agent SDK, with a Cursor API key, next to the other harnesses ([Cursor on Replicas](https://docs.replicas.dev/features/coding-agents/cursor)). We gave it the same humanize bug from our first test in a Replicas workspace. It fixed the bug, added eight regression cases and the full test suite passed (770 tests).

| | One harness (Devin, Cursor) | Several harnesses on one platform (Replicas) |
|---|---|---|
| Who tunes the harness | The vendor tunes its own harness for the models it offers. Cursor says its harness is "heavily customized for every model we support" ([Cursor](https://cursor.com/blog/continually-improving-agent-harness)) | Each harness's vendor. Replicas adds its own instructions and tools to each session but doesn't tune a harness for a model |
| Switching harness or model | Switch models inside that one harness | Pick a different harness or model for each chat. The environment, repositories, secrets and start-up commands stay the same ([environments](https://docs.replicas.dev/features/environments)) |
| Automations | Run on that product's agent | Each automation can set its own harness, model and effort, and falls back to the organization's default ([automations](https://docs.replicas.dev/features/automations)) |
| Sign-in and billing | One account and one bill for the agent | A sign-in for each harness you use, such as a Claude plan, a ChatGPT sign-in or a Cursor API key, plus the platform's own charge |
| Features | One feature set, built for that agent | Features differ by harness. For example, Cursor on Replicas doesn't support Fast Mode or goal mode ([Cursor on Replicas](https://docs.replicas.dev/features/coding-agents/cursor)) |

The trade-off is between depth and choice. A single-harness product is designed end to end: the vendor shapes its prompts, tools and machine for its own agent and the models it offers. A platform that runs several harnesses lets you move to a different harness or model when one is cheaper, faster or better at a kind of task, without moving your environment or automations, but it doesn't tune any harness itself, and you manage more sign-ins.

## Running several harnesses on Replicas, and its limits

Replicas runs each task in its own cloud VM with your repository checked out, so you can send the same task to two harnesses and compare the pull requests. Here is what to know before you do:

- **You bring each harness's sign-in.** For example, Claude Code takes a Claude plan or an Anthropic API key, and Codex a ChatGPT sign-in or an OpenAI API key. Your model provider bills the tokens, and Replicas bills the VM by the minute: $0.004 to $0.016 a minute while it's awake, by size ([Replicas billing](https://docs.replicas.dev/admin/billing)).
- **OpenCode and Pi have no Anthropic key option there.** On Replicas they connect through OpenCode Go, Aster, OpenRouter, an OpenAI-compatible endpoint or Ollama ([Replicas docs](https://docs.replicas.dev/admin/credentials)), so the Claude Sonnet 5.5 runs above would go through a provider such as OpenRouter there.
- **Approval prompts are off.** Claude Code, Codex, OpenCode and Pi run with their approval checks turned off inside the VM, so the VM is the safety boundary, not the harness's permission settings. Admins can still block agents from merging pull requests or pushing to protected default branches ([security settings](https://docs.replicas.dev/admin/security)). Give each environment only the secrets its tasks need.
- **Replicas adds instructions.** Each session gets the environment's system prompt ([environments](https://docs.replicas.dev/features/environments)) and Replicas' own workspace instructions, so its requests are larger than the bare runs above.

To run the same task through several harnesses on your own repository, [try Replicas with $100 of free usage](https://app.replicas.dev).


## FAQ

### What is an agent harness?
The software around a language model that turns it into an agent. It runs the loop that sends the model the conversation and a list of tools, carries out the tool calls the model asks for, decides what goes into the context window, and enforces what the agent may do. Claude Code, Codex, OpenCode and Pi are agent harnesses for coding.

### Is Claude Code an agent or a harness?
Claude Code is a harness. It supplies the loop, the tools, the instructions and the permission rules, and a Claude model does the reasoning. A Claude Code session working on a task is an agent. VS Code uses the word the same way: its chat can run the Copilot, Claude or Codex harness.

### What is the difference between an agent harness and an agent framework?
A framework is a library you use to build a harness, such as Microsoft Agent Framework. A harness is the program that runs a model in a loop with tools and rules, such as Claude Code or Pi.

### Does the harness change how much a coding agent costs?
Yes. In our test, the same bug fix with the same model cost about 2.4 times as much in the most expensive harness as in the cheapest with Claude Sonnet 5.5, and about 2.1 times as much with GPT-6.2-Sol. The harnesses sent different amounts of fixed prompt, made different numbers of requests and asked for different reasoning effort. A study of seven models in three harnesses found cost differences from under 10% to three or four times, depending on the model.

### Can you use Cursor in Replicas?
Yes. Replicas runs Cursor's own harness inside the workspace through the Cursor Agent SDK, with a Cursor API key, alongside Claude Code, Codex, Gemini, Muse Code, OpenCode and Pi. You pick the harness for each chat or automation. Some Replicas features, such as Fast Mode and goal mode, don't work with Cursor.

### What is harness engineering?
The work of building the part of a coding agent's harness that your team adds for its own codebase: AGENTS.md files, skills, linters, tests and hooks that steer the agent and check its output. OpenAI used the term in February 2026 to describe building a product with Codex.
