Search for a command to run...
No comments yet. Be the first to comment.
A hands-on series exploring AI harnesses, agent systems, and the engineering patterns that make AI agents more capable and reliable. I’ll break down existing harnesses, build my own, and experiment with small models to understand what actually works.
An LLM can write code.
But writing code isn't the hard part.
The hard part is getting the model to keep working when the task takes hundreds of steps, the context gets messy, the code doesn't behave as expected, and the model confidently tells you everything is fine.
That's where the harness comes in.
At least, that's what I thought going in.
I've spent the last couple of days trying to actually understand this thing. Read through a handful of articles and a research paper, went through some GitHub repos around it too.
This isn't a definitional blog on harnesses. It's just my understanding of what a harness is, so feel free to correct me if I got something wrong.
Digging into Claude Code, I found that they originally had an initializer agent. It set the starting point, wrote out a feature_list.md, handled system prompts, and then handed off to a coding agent that actually picked up the task and completed it, moving to the next task only once the current one satisfied the evidence for being done.
This worked, but context anxiety and the model's own bias toward rating its work favorably made it fall apart on longer tasks. So they introduced a three-agent setup: Planner, Generator, Evaluator, inspired by GAN architecture.
So: is that above setup a "harness," by Anthropic's own claim? Yes. The system built around the model to get better results than a bare LLM call, that's what a harness is.
It's genuinely different from prompt and context engineering. I do think prompt and context engineering are part of a harness, but not the whole thing. Not the full body of it. There are other requirements, laid out well in this paper: What Makes a Harness a Harness (paper)
They lay out four minimum requirements for a system to be called a harness:
Agent loop. The core reasoning-action-observation cycle: query the model, act on what it returns, feed the result back in, repeat until the task's done or a stop condition fires.
Tools / environment. This can be anything: web, bash tools, MCPs, etc.
Context management. How much, and which, tokens get passed to the LLM to generate output. (Same point as above, context is part of a harness, not the whole of it.)
Control mechanism. Step limits, stop-on-error conditions, retry limits, guardrails.
If all four are satisfied, it's a harness. Miss any one, and it isn't.
There are extras on top of this too: model-switch retries, memory, and other things that make a harness more capable and complex. They're just not part of the minimum bar.
A simple harness that satisfies all four is mini-swe-agent. Let's see how.
1. Loop:
def run(self, task: str = "", **kwargs) -> dict:
"""Run step() until agent is finished. Returns dictionary with exit_status, submission keys."""
self.extra_template_vars |= {"task": task, **kwargs}
self.messages = []
self.add_messages(
self.model.format_message(role="system", content=self._render_template(self.config.system_template)),
self.model.format_message(role="user", content=self._render_template(self.config.instance_template)),
)
while True:
try:
self.step()
self.n_consecutive_format_errors = 0 # reset on any clean step
except FormatError as e:
self.cost += e.messages[0].get("extra", {}).get("cost", 0.0)
self.n_consecutive_format_errors += 1
if 0 < self.config.max_consecutive_format_errors <= self.n_consecutive_format_errors:
self.add_messages(
*e.messages,
{
"role": "exit",
"content": "RepeatedFormatError",
"extra": {"exit_status": "RepeatedFormatError", "submission": ""},
},
)
else:
self.add_messages(*e.messages)
except InterruptAgentFlow as e:
self.add_messages(*e.messages)
except Exception as e:
self.handle_uncaught_exception(e)
raise
finally:
self.save(self.config.output_path)
if self.messages[-1].get("role") == "exit":
break
return self.messages[-1].get("extra", {})
run() has a while True loop that calls step(). step() calls query(), which passes the message history to the LLM and gets output back, usually a tool call, and that tool call gets executed.
2. Tools / environment:
def execute_actions(self, message: dict) -> list[dict]:
"""Execute actions in message, add observation messages, return them."""
outputs = [self.env.execute(action) for action in message.get("extra", {}).get("actions", [])]
return self.add_messages(*self.model.format_observation_messages(message, outputs, self.get_template_vars()))
execute_actions() takes the actions from the model's message and executes them based on whatever environment is configured: local subprocess, Docker exec, Singularity/Apptainer. Swapping environments only means swapping what self.env.execute() points to, nothing else in the code needs to change.
3. Context management:
def query(self) -> dict:
"""Query the model and return model messages. Override to add hooks."""
if 0 < self.config.step_limit <= self.n_calls or 0 < self.config.cost_limit <= self.cost:
raise LimitsExceeded(...)
if 0 < self.config.wall_time_limit_seconds <= int(time.time() - self._start_time):
raise TimeExceeded(...)
self.n_calls += 1
message = self.model.query(self.messages)
self.cost += message.get("extra", {}).get("cost", 0.0)
self.add_messages(message)
return message
self.messages is the context passed to the model. At this level, mini-SWE-agent maintains the conversation history as the agent progresses; whether that constitutes the paper's stronger notion of active context management depends on how the model/history is actually selected, trimmed, or transformed during execution.
4. Control:
There's no dedicated method for this. It's enforced through the same if/else checks already visible in query() above:
if 0 < self.config.step_limit <= self.n_calls or 0 < self.config.cost_limit <= self.cost:
raise LimitsExceeded(...)
if 0 < self.config.wall_time_limit_seconds <= int(time.time() - self._start_time):
raise TimeExceeded(...)
Plus:
if self.messages[-1].get("role") == "exit":
break
Together, these don't make the agent deterministic, but they make its execution bounded and give the harness deterministic stopping conditions: hitting a limit, timing out, or actually finishing the work.
By this definition, mini-SWE-agent satisfies all four conditions, at least on a loose reading of context management, so I'd classify it as a harness.
Depends on the model underneath it. On the mini-swe-agent leaderboard, Claude Opus 4.5 (high) scores 76.80%, with Gemini 3 Flash (high) close behind at 75.80%. GPT-4o sits far lower, around 21.62%. The model clearly matters. But these numbers are also a reminder that we're not measuring the model in isolation, we're measuring the model running inside a specific agent system, since SWE-bench Verified uses mini-SWE-agent as the common harness for these evaluations. That makes the leaderboard useful for comparing model-plus-harness systems, but it doesn't cleanly separate how much of the performance comes from the model versus how much comes from the harness itself.
I tested it myself with a smaller model, Qwen3.5 4B, running locally on Ollama. It handled simple tasks fine. It failed on something slightly harder: building a Flappy Bird game. It failed on the physics, the bird crashed while passing through the poles, even though it should have made it through cleanly.
But I don't think the failure was purely a model problem. Part of it was the harness.
You could argue: if Gemini 3 does fine with the same harness, why doesn't Qwen3.5? Fair enough, it's a much larger model, obviously it'll do better. But we're not here to compare models. We're here to look at the harness. And a bash-only harness like this one is missing something specific: vision.
or more precisely, any real observation channel into what the application actually does once it's running.
Right now the harness can tell the model "the script ran without an error." It can't tell the model whether the bird actually cleared the pipe or crashed into it. That's the gap. If the model gets fed real evidence that its code is failing, not just confirmation that it executed, it can likely course-correct into something much better than one-shotting the physics blind. Something like Puppeteer or Playwright MCP would let the model test its own output, understand a screenshot, see where it actually failed, and fix it with that in hand.
Harnesses aren't limited to coding agents, either. They can wrap around anything an agent is capable of doing. But the harness itself has to shift a lot depending on the application. The right harness depends on where the model fails.
If the model loses track of long tasks, give it better context management. If it can't verify its own work, give it an evaluator. If it can't see the application's output, give it an observation channel. If it runs forever, give it control mechanisms.
The interesting part isn't building a bigger harness. It's figuring out exactly what the model needs help with, and engineering only that part around it.
I'm going to try building my own harness from scratch, leaning heavily on mini-swe-agent as a base since it's the simplest real harness out there. I'll also look at Pi and OpenHands for inspiration, though both are more specialized toward particular use cases.
Then I'll try the Flappy Bird task again, one-shot, with a harness, on the same small model (Qwen3.5 4B). My focus going forward is two things:
Getting better results out of smaller models through the harness itself.
Building a focused "friendly neighborhood harness," one that handles a specific problem and does that job properly, rather than trying to be general-purpose.
This is my first proper technical post, so a quick introduction: I'm a fresher AI engineer, and I'm using this blog to document what I actually build, test, and break while learning in public. I'm still figuring a lot of this out, so if I've misunderstood something, I'd genuinely like to hear about it. Follow along on X, and subscribe to the blog if that sounds useful. More of these coming.