All resources
Guide/8 min read

What is an agent harness? Meaning, examples and a four-harness test

Published

Summary

An agent harness is the software that turns a model into an agent: its loop, tools, context and permissions. We tested Claude Code, Codex, OpenCode and Pi.

An agent harness is the software around a language model that turns it into an agent. It runs the loop that sends the model a task and a list of tools, carries out the tool calls the model asks for, decides what goes into the model's context, and enforces what the agent is allowed to do. Claude Code, Codex, OpenCode and Pi are agent harnesses for coding.

The model on its own only reads text and writes text back. The harness is what lets it edit files and run commands. LangChain sums it up as "Agent = Model + Harness" (LangChain). Other names for it are agentic harness, AI agent harness and agent scaffolding (Wikipedia). "LLM harness" is used for it too, but also for evaluation tools that run benchmarks against a model, such as EleutherAI's lm-evaluation-harness.

This guide explains what a harness contains and how it differs from a model, a framework and an agent. Then it reports tests we ran on November 22, 2026: we gave the same coding task to four harnesses, three times for each harness and model, and held the model constant to see what the harness changes.

What an agent harness does

Most definitions list the same parts (Wikipedia, LangChain, VS Code). The last column shows what each part looked like in our test.

PartWhat it doesIn our test
Agent loopSends the conversation to the model, runs the tool calls it returns, adds the results and sends it again, until the model stops4 to 11 model requests for the same task
ToolsThe actions the model can ask for, such as reading and editing files, running shell commands, searching the web or starting a subagent4 tools in Pi, 11 in Claude Code plus 13 more it loads on request
InstructionsA fixed system prompt, details about the machine, and project files such as AGENTS.md1,343 to 17,698 tokens before the model had read any code
Context managementWhat stays in the context window as a session grows: prompt caching, trimming old output, and summarizing old turns (compaction)Later sessions read part or all of the fixed prompt from the provider's cache
Permissions and sandboxWhich actions run without asking, which need approval, and what the process can reachWith default settings, two of the four harnesses changed nothing
Model settingsWhich model to call, how much reasoning effort to ask for, and the output limitClaude Sonnet 5.5 ran at low, medium or high effort depending on the harness

Agent harness vs model, framework and agent

These four terms are often mixed up. A model can run in many harnesses, and many harnesses can run more than one model.

TermWhat it isExamples
ModelThe language model. It has no memory between requests and can't act on its ownClaude Sonnet 5.5, GPT-6.2-Sol
Agent harnessThe program that runs a model in a loop with tools, instructions and rulesClaude Code, Codex, OpenCode, Pi
Agent frameworkA library for building your own harnessMicrosoft Agent Framework (Microsoft Learn)
AgentA harness running a model on a taskClaude Code with Claude Sonnet 5.5 fixing a bug

So Claude Code is a harness, and a Claude Code session working on a task is an agent. VS Code's docs use the word the same way: its chat can run the Copilot, Claude or Codex harness, and the harness you pick decides the tools, the models on offer and the approval modes (VS Code).

What is harness engineering?

Harness engineering is the work of building the parts of a coding agent's harness that your team controls, so the agent does reliable work in your codebase. OpenAI introduced the term in February 2026 in a post about building a product with Codex and "0 lines of manually-written code". It describes the engineer's job as to "design environments, specify intent, and build feedback loops" (OpenAI).

Birgitta Böckeler explains how it relates to the harness itself (martinfowler.com). Part of a coding agent's harness is already built in, such as its system prompt and tools, and agents also let users build an "outer harness" for their own codebase. She calls that work "a specific form of context engineering", and splits it into two kinds of control:

Kind of controlWhat it doesExamples
GuidesSteer the agent before it actsAn AGENTS.md file with your conventions, skills
SensorsCheck the result after it acts, so the agent can correct itselfLinters, type checkers, structural tests in a pre-commit hook, AI code review

OpenAI's post gives concrete examples of both: an AGENTS.md kept short as "the table of contents" for a docs/ folder in the repository, and architecture rules "enforced mechanically via custom linters" and structural tests.

Our test used the smallest possible guide: a two-line AGENTS.md asking agents to log every fix in CHANGES.md. All four harnesses loaded it, and every run followed it, even though the prompt never mentioned the file. The rest of this guide is about the built-in part of the harness, which you choose rather than build.

The same task through four harnesses

We used a real bug in humanize, an MIT-licensed Python library: ordinal(-1) returns -1th instead of -1st. Each harness got the same prompt in a fresh copy of the repository, and the repository's AGENTS.md asked agents to record every fix in CHANGES.md. The prompt didn't mention that file, so a changelog line shows the harness had passed AGENTS.md to the model.

We ran Claude Code with Claude Sonnet 5.5 and Codex with GPT-6.2-Sol, and ran OpenCode and Pi with both models. That gives two groups where the model is the same and only the harness changes. How we tested has the details.

Cost and time per run, same task and same model

Median of three runs per harness on November 22, 2026. Cost at list prices

Claude Sonnet 5.5

Claude Code

4 model requests · first request 17,698 tokens

Cost

$0.067

Time

20 s

OpenCode

6 model requests · first request 10,779 tokens

Cost

$0.062

Time

28 s

Pi

4 model requests · first request 2,638 tokens

Cost

$0.028

Time

20 s

GPT-6.2-Sol

Codex

6 model requests · first request 11,756 tokens

Cost

$0.062

Time

29 s

OpenCode

11 model requests · first request 5,037 tokens

Cost

$0.051

Time

37 s

Pi

7 model requests · first request 1,343 tokens

Cost

$0.030

Time

24 s

The first request is everything the harness sent before the model had looked at any code: its system prompt, tool definitions, AGENTS.md, environment details and our prompt. Anthropic and OpenAI count tokens differently, so compare token counts within a model group, not across groups. Costs use each provider's list prices (Anthropic, OpenAI).

  • All 18 runs did the job. Every run fixed the bug, added regression tests, passed the full test suite and added the changelog line that AGENTS.md asked for.
  • Cost changed more than the result. With the same model, the most expensive harness cost about 2.4 times as much per run as the cheapest with Claude Sonnet 5.5, and about 2.1 times as much with GPT-6.2-Sol.
  • Pi was the cheapest with both models. It sends the shortest prompt and only four tools, and it asked Claude Sonnet 5.5 for low effort. Claude Code asked for medium effort, and OpenCode set no effort, so the API default of high applied (Anthropic).
  • This was one small task. A bigger prompt or more tools can pay off on harder work, and this test can't show that.

A second task, with reasoning effort set to high

The harnesses asked for different reasoning effort in the first test, so we ran a second task with each harness set to high effort: --effort high in Claude Code, model_reasoning_effort=high in Codex, --variant high in OpenCode and --thinking high in Pi. The task was a small Node.js project we wrote for the test. One test failed because tax was charged before the discount, and the agent had to fix that, add a BULK10 discount code with tests, and document it. We checked every run against seven hidden tests the agents never saw. This time the Anthropic model was Claude Opus 5.5.

Model and harnessTime per runModel requestsFirst request, in tokensCost per run
Claude Opus 5.5 in Claude Code20 s417,559$0.160
Claude Opus 5.5 in OpenCode23 s510,639$0.128
Claude Opus 5.5 in Pi19 s52,474$0.071
GPT-6.2-Sol in Codex25 s714,919$0.078
GPT-6.2-Sol in OpenCode36 s124,959$0.054
GPT-6.2-Sol in Pi21 s61,252$0.026

All 18 runs passed the hidden tests and the project's original tests. Pi was again the cheapest with both models, and the gap between the cheapest and most expensive harness was about 2.3 times on Claude Opus 5.5 and 3 times on GPT-6.2-Sol. As in the first test, we priced each run's first request as if the cache were empty. Claude Code's runs actually read about 12,000 tokens from a cache that earlier sessions on the same API account had warmed, which brought its median to $0.103. With that cache hit, it cost less than OpenCode.

What changed between the harnesses

Tools

HarnessTools in the first request
Claude Code11: Agent, Bash, Edit, ListAgents, Read, ReportFindings, ScheduleWakeup, Skill, ToolSearch, Workflow, Write. It also listed 13 more by name, such as WebFetch and WebSearch, which load only when the model asks for them through ToolSearch
Codex9: exec_command, write_stdin, request_user_input, view_image, multi_agent_v1, get_goal, create_goal, update_goal, web_search. It edits files with apply_patch run through the shell. In the second task, with high effort and its workspace-write sandbox, Codex instead gave the model an exec tool that runs JavaScript, which called exec_command and apply_patch, sometimes several at once
OpenCode10 with Claude Sonnet 5.5: bash, edit, glob, grep, read, skill, task, todowrite, webfetch, write. With GPT-6.2-Sol, apply_patch replaced edit and write
Pi4: read, bash, edit, write

These lists come from the requests each harness sent in headless mode with no MCP servers or skills installed. Interactive sessions and extra servers change them. For a tool-by-tool comparison of Claude Code and OpenCode, see OpenCode vs Claude Code.

Context and instructions

HarnessReads AGENTS.mdRead from the previous session's cache in our runs
Claude CodeYes, when the repository has no CLAUDE.md (Claude Code docs)11,348 tokens of its prompt
CodexYes (Codex docs)None
OpenCodeYes, and reads CLAUDE.md where there is no AGENTS.md (OpenCode docs)With Claude Sonnet 5.5, all 10,754 tokens when the next session ran in the same folder, and none in a new folder. With GPT-6.2-Sol, none
PiYes, and CLAUDE.md files too (Pi docs)1,310 tokens with Claude Sonnet 5.5. None with GPT-6.2-Sol

All four harnesses put the AGENTS.md text into their first model request, and every run followed it.

Caching is why a large fixed prompt costs less than its size suggests. When a later request starts with the same text, the provider reads it from its cache at a twentieth of the normal input price (Anthropic, OpenAI). Within one session, every harness got cache hits from the second request on. Between sessions, it depends on whether the start of the prompt changes. OpenCode puts the working folder into the cached part of its prompt, so a new folder means a new cache entry. With GPT-6.2-Sol, no harness got a cache hit on a session's first request, and all three sent a different prompt_cache_key for each session.

None of our runs came close to a context limit. The largest request was about 22,000 tokens, so this test says nothing about how each harness summarizes a long session.

Permissions

We also ran each harness once with no permission flags, as a script or CI job would.

HarnessCommandWhat happened with default settingsFlag we used for the main runs
Claude Codeclaude -pStarted in Manual mode. Claude Code denied the file edits, changed nothing and described the fix it would have made (Claude Code docs)--dangerously-skip-permissions
Codexcodex execRan in a read-only sandbox. Codex tested the fix in memory, printed a diff and changed nothing (Codex docs)--dangerously-bypass-approvals-and-sandbox
OpenCodeopencode runMade the fix. OpenCode's docs describe a default policy that only asks before reading .env files (OpenCode docs), but in the second task's probe opencode run 1.19.4 still rejected a write outside the project--auto
Pipi -pMade the fix. Pi has no approval step for tool calls and no built-in sandbox (Pi docs)None needed

In the second task we used narrower settings and asked each harness to write a file outside the project and fetch a web page. Codex's workspace-write sandbox blocked both, because it turns off network access (Codex docs). Claude Code in acceptEdits mode blocked the outside write, and fetched the page only because we had allowed Bash. OpenCode's run rejected the outside write and fetched the page. Pi did both.

The flags in the last column turn off the checks, so only use them where the agent can't reach anything you'd mind losing. How to run Claude Code in a sandbox compares the safer ways to do that.

Model settings

The model was the same within each group, but the harnesses called it differently.

HarnessReasoning effort it asked forOutput limitExtra requests
Claude Code, with Claude Sonnet 5.5Medium128,000 tokensNone
OpenCode, with Claude Sonnet 5.5None set, so the API default of high32,000 tokensOne request to Claude Haiku 5.5 to title the session
Pi, with Claude Sonnet 5.5Low128,000 tokensNone
Codex, with GPT-6.2-SolNone setNot setNone
OpenCode, with GPT-6.2-SolNone setNot setOne request to GPT-5.4 nano to title the session
Pi, with GPT-6.2-SolMedium128,000 tokensNone

Each harness lets you set the reasoning effort, for example with --effort in Claude Code, --variant in OpenCode and --thinking in Pi. In the first task we left every harness at its defaults, because that is what most people run.

Does the harness change the result?

In our tests it didn't: all 36 runs passed. A larger study came to a similar conclusion. Yangze Liu and Zhongyi Han ran seven models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified and two other benchmarks (arXiv). No harness led across all models. Where they could measure it, the gap in pass rate was 2 to 5 points, and swapping the harness changed about as many task outcomes as rerunning the same one. Cost differences between harnesses ranged from under 10% to three or four times, depending on the model.

So when you choose a harness, the cost per task, the speed, the default permissions and how well it fits your tools matter as much as benchmark scores. These pages compare specific harnesses:

How we tested

An AI agent working in a Replicas workspace set up and ran these tests on November 22, 2026, and a person on our team reviewed this guide before it was published. Replicas is our product.

  • Machine. One Replicas workspace, a Linux VM. We ran each harness's own command-line tool in headless mode, not through the Replicas chat, so Replicas' own instructions aren't in these numbers.
  • Versions. Claude Code 2.1.330, Codex CLI 0.167.0, OpenCode 1.19.4 and Pi 1.6.0.
  • Models. Claude Sonnet 5.5 with an Anthropic API key, and GPT-6.2-Sol with an OpenAI API key.
  • Repository. humanize at commit a950045, plus a two-line AGENTS.md and an empty CHANGES.md. There was no customer code or data.
  • Setup. Every run started from a fresh clone in a new folder and an empty home folder, so no user settings, skills or MCP servers were loaded. For the main runs we turned approvals off with each harness's flag. We ran one warm-up round, then three measured runs of each pairing.
  • Measurement. A local proxy recorded every request each harness sent and the token counts the API returned. Time is wall-clock time from start to exit.
  • Checks. We compared ordinal() against the correct suffix for every integer from -300 to 300, ran the full test suite, and checked CHANGES.md for the new line.
  • Cost. Each run's cost uses the token counts the API returned and the list prices on the day. Because every run used a new folder, we priced each run's first request as if the cache were empty, so a harness wasn't penalized for having the folder in its cached prompt.
  • A mistake we fixed. In our first Codex runs, Codex's home folder sat under /tmp, where Codex won't install its apply_patch helper, so it spent extra requests working around that. We moved the home folder and ran Codex again. The table shows the second set.

The prompt we gave every harness:

The second task ran the same way: Claude Code 2.1.330, Codex CLI 0.167.0, OpenCode 1.19.4 and Pi 1.6.0, a fresh copy of the project and an empty home folder for every run, three runs per pairing, effort set to high, and costs from the token counts each harness reported. Its Codex runs kept the home folder under /tmp. Codex still applied its patches, but its numbers may differ from a normal install.

For the Cursor example, we started one Replicas workspace with Cursor's default model through the Replicas CLI (replicas create -a cursor), in an environment with no repositories, and gave it the first task's prompt plus a step to clone humanize. We ran it once and didn't measure its cost.

These are two small tasks run three times per pairing, not a benchmark. We didn't test long sessions, compaction, MCP servers, interactive mode, subscription sign-ins, or how each harness handles a task it can't finish.

One harness, or several on one platform

Some coding agent products are a single harness built around their own agent. Devin's docs describe one agent, Devin, working in a Linux virtual machine "with your repositories cloned, tools installed, dependencies resolved" (Devin docs). Cursor runs its own agent harness in its editor, its CLI and its Cloud Agents, which "run in isolated VMs in the cloud" with "a curated selection of models" (Cursor docs). In both products, the company that makes the harness also runs the machine it works in.

Other platforms run several harnesses in the same place. Replicas is one of them: Claude Code, Codex, Cursor, Gemini (through Google's Antigravity CLI), Muse Code, OpenCode and Pi each run as their vendor ships them, inside the same kind of cloud VM (Replicas docs). Cursor is a good example of the difference. In Replicas, Cursor's own harness runs inside the workspace through the Cursor Agent SDK, with a Cursor API key, next to the other harnesses (Cursor on Replicas). We gave it the same humanize bug from our first test in a Replicas workspace. It fixed the bug, added eight regression cases and the full test suite passed (770 tests).

One harness (Devin, Cursor)Several harnesses on one platform (Replicas)
Who tunes the harnessThe vendor tunes its own harness for the models it offers. Cursor says its harness is "heavily customized for every model we support" (Cursor)Each harness's vendor. Replicas adds its own instructions and tools to each session but doesn't tune a harness for a model
Switching harness or modelSwitch models inside that one harnessPick a different harness or model for each chat. The environment, repositories, secrets and start-up commands stay the same (environments)
AutomationsRun on that product's agentEach automation can set its own harness, model and effort, and falls back to the organization's default (automations)
Sign-in and billingOne account and one bill for the agentA sign-in for each harness you use, such as a Claude plan, a ChatGPT sign-in or a Cursor API key, plus the platform's own charge
FeaturesOne feature set, built for that agentFeatures differ by harness. For example, Cursor on Replicas doesn't support Fast Mode or goal mode (Cursor on Replicas)

The trade-off is between depth and choice. A single-harness product is designed end to end: the vendor shapes its prompts, tools and machine for its own agent and the models it offers. A platform that runs several harnesses lets you move to a different harness or model when one is cheaper, faster or better at a kind of task, without moving your environment or automations, but it doesn't tune any harness itself, and you manage more sign-ins.

Running several harnesses on Replicas, and its limits

Replicas runs each task in its own cloud VM with your repository checked out, so you can send the same task to two harnesses and compare the pull requests. Here is what to know before you do:

  • You bring each harness's sign-in. For example, Claude Code takes a Claude plan or an Anthropic API key, and Codex a ChatGPT sign-in or an OpenAI API key. Your model provider bills the tokens, and Replicas bills the VM by the minute: $0.004 to $0.016 a minute while it's awake, by size (Replicas billing).
  • OpenCode and Pi have no Anthropic key option there. On Replicas they connect through OpenCode Go, Aster, OpenRouter, an OpenAI-compatible endpoint or Ollama (Replicas docs), so the Claude Sonnet 5.5 runs above would go through a provider such as OpenRouter there.
  • Approval prompts are off. Claude Code, Codex, OpenCode and Pi run with their approval checks turned off inside the VM, so the VM is the safety boundary, not the harness's permission settings. Admins can still block agents from merging pull requests or pushing to protected default branches (security settings). Give each environment only the secrets its tasks need.
  • Replicas adds instructions. Each session gets the environment's system prompt (environments) and Replicas' own workspace instructions, so its requests are larger than the bare runs above.

To run the same task through several harnesses on your own repository, try Replicas with $100 of free usage.

FAQ

Frequently asked questions