In this post I'll cover where Sadhan is right now, the difficulties I hit, how I solved them, what I learned along the way, and what I plan to do next.

Version 1: The Simple Loop

The first version of the harness was small. I gave it a bash tool that ran a single command, and the bash process closed after each one, so nothing persisted between commands.

The flow worked like this:

1. The agent loop starts and initializes the state.

2. The user's query is added to the state and passed to the LLM.

3. The LLM returns a message with <reasoning> and <action> tags.

4. A parser method pulls out the reasoning and action and returns them as JSON.

5. The action part is executed as a bash command.

6. The return code and output from the bash tool go back to the LLM, and the loop continues.

That is the whole idea. The model thinks, picks an action, the harness runs it, and the result goes back so the model can decide what to do next. This worked well for a while. I used it for single-file coding tasks and it handled them fine, so at this point I felt the hard part was behind me. It wasn't.

Problem 1: Bash With No Memory

Single-command bash breaks down quickly on real projects. Anything that depends on earlier commands, like working inside a Python venv or moving through nested directories, became really difficult without a persistent shell.

Think about a normal workflow. You cd into a folder, activate a virtual environment, and then run your tests. Each of those steps depends on the one before it. When every command runs in a fresh shell, the agent keeps losing its place, and the model has to work around a limitation that a human developer never has to think about.

So I replaced it with an asyncio-based persistent shell, where one shell process stays alive for the whole session and every command runs inside it.

Problem 2: How Do I Know a Command Has Finished?

A persistent shell created a new problem. Before, a command ended when its process ended, so knowing it was done was free. Now the shell never exits. When the LLM sends a command, how do I know it has completely finished and isn't still running? Commands like pip install make this obvious, since they produce output over a long stretch of time. If I return too early, the model sees half the output and makes decisions on incomplete information.

My fix was a sentinel marker. After every LLM-given command, I append an echo of a unique marker plus the exit code:

python

marker = f"__SADHAN_DONE_{token}_"
payload = f"{command}\necho {marker}$?\n"

The shell runs the command, and then prints the marker followed by the exit code of that command. I read the output stream until the marker shows up. When it does, I know the command is fully done, and I get the return code from the same line.

This change gave the agent much better power. It can now move around nested directories and work across multi-file projects, because the shell keeps its state from one command to the next.

Problem 3: Tag-Based Output and Small Models

The tag-based LLM output gave me a lot of headaches. Small models like Qwen 3.5 9B would often write the opening <action> tag but never the closing </action>. That caused a lot of parsing errors, and many times the model just gave me broken results instead of something the harness could run.

I could have kept making the parser more forgiving, but that felt like patching a symptom. So I switched to tool-based calling. I know small models are not great at tool calling, but I tried it as an experiment and it gave me better results. It still isn't always correct, but it is a lot better than the tag-based approach.

Supporting More Models

Until this point I had only tried local models, and there was no model registry for anything else. I added LiteLLM as the provider layer so I can switch models without touching the core loop. It doesn't cover every model yet, but the popular ones like Gemini, OpenAI, and Anthropic are in.

Sessions and Modes

I also added session management. Conversations are stored by directory path and time inside .sadhan/sessions/. This makes it easy to pick up a project I started last week without starting over, and sessions can be renamed for convenience.

How Sadhan Works Today

1. The user installs it in a venv with pip install sadhan and runs sadhan.

2. A TUI opens, and the user types the task.

3. The state's message array gets the system prompt first, then the user's task.

4. A while True loop runs. The messages go to the LLM, which reasons over them and returns a tool call if needed.

5. The tool call is executed in the persistent shell and the output goes back to the LLM.

6. This repeats until the task is done.

Stopping conditions: the loop can't run forever. If the same kind of error happens 5 times in a row, the agent terminates automatically and kills all child processes, so nothing is left running in the background. It also terminates when the default step limit is exhausted.

What I Learned

The loop is the easy part. The first version was basically LLM, tool, result, repeat, and it came together fast. Almost every real problem showed up around the loop: shell state, knowing when a command is done, the model's output format, and how to stop safely.

Persistence changes everything. Moving from one-shot bash to a persistent shell wasn't a small upgrade. It made the agent usable on real projects, but it also brought a new class of problems, like detecting completion, that I never had to think about before.

Keep the model layer swappable. Using LiteLLM means the loop doesn't care which model is behind it. That matters when you are experimenting with local models and hosted ones side by side.

Small usability features matter. Sessions look minor, but being able to come back to a project a week later without starting over changes how the tool feels to use.

An agent needs a way to stop. Without the 5 consecutive errors rule and the step limit, a confused model can keep looping and leave processes running. Termination and cleanup need to be designed in from the start.

Building it is not the same as knowing it works. I've been testing by feel so far. That is why evaluation is the next big thing on my list.

What's Next

• Evaluation. I haven't properly evaluated the agent yet. In a couple of weeks I'll run it on 3 to 4 real GitHub repos to fix bugs or add features, and share the results.

• Plan mode. Along with an evaluator mode that runs in the backend and isn't used directly by the user.

• Multimodal input. Right now it only understands text. I want to make it handle images and other data later.

Sadhan is still a work in progress, and I'll keep sharing what I learn as it grows. More updates soon.

An LLM can write code.

But writing code isn't the hard part.

The hard part is getting the model to keep working when the task takes hundreds of steps, the context gets messy, the code doesn't behave as expected, and the model confidently tells you everything is fine.

That's where the harness comes in.

At least, that's what I thought going in.

I've spent the last couple of days trying to actually understand this thing. Read through a handful of articles and a research paper, went through some GitHub repos around it too.

This isn't a definitional blog on harnesses. It's just my understanding of what a harness is, so feel free to correct me if I got something wrong.

What Anthropic actually built

Digging into Claude Code, I found that they originally had an initializer agent. It set the starting point, wrote out a feature_list.md, handled system prompts, and then handed off to a coding agent that actually picked up the task and completed it, moving to the next task only once the current one satisfied the evidence for being done.

This worked, but context anxiety and the model's own bias toward rating its work favorably made it fall apart on longer tasks. So they introduced a three-agent setup: Planner, Generator, Evaluator, inspired by GAN architecture.

So: is that above setup a "harness," by Anthropic's own claim? Yes. The system built around the model to get better results than a bare LLM call, that's what a harness is.

It's genuinely different from prompt and context engineering. I do think prompt and context engineering are part of a harness, but not the whole thing. Not the full body of it. There are other requirements, laid out well in this paper: What Makes a Harness a Harness (paper)

The four conditions

They lay out four minimum requirements for a system to be called a harness:

1. Agent loop. The core reasoning-action-observation cycle: query the model, act on what it returns, feed the result back in, repeat until the task's done or a stop condition fires.

2. Tools / environment. This can be anything: web, bash tools, MCPs, etc.

3. Context management. How much, and which, tokens get passed to the LLM to generate output. (Same point as above, context is part of a harness, not the whole of it.)

4. Control mechanism. Step limits, stop-on-error conditions, retry limits, guardrails.

If all four are satisfied, it's a harness. Miss any one, and it isn't.

There are extras on top of this too: model-switch retries, memory, and other things that make a harness more capable and complex. They're just not part of the minimum bar.

Checking it against mini-swe-agent

A simple harness that satisfies all four is mini-swe-agent. Let's see how.

1. Loop:

def run(self, task: str = "", **kwargs) -> dict:
    """Run step() until agent is finished. Returns dictionary with exit_status, submission keys."""
    self.extra_template_vars |= {"task": task, **kwargs}
    self.messages = []
    self.add_messages(
        self.model.format_message(role="system", content=self._render_template(self.config.system_template)),
        self.model.format_message(role="user", content=self._render_template(self.config.instance_template)),
    )
    while True:
        try:
            self.step()
            self.n_consecutive_format_errors = 0  # reset on any clean step
        except FormatError as e:
            self.cost += e.messages[0].get("extra", {}).get("cost", 0.0)
            self.n_consecutive_format_errors += 1
            if 0 < self.config.max_consecutive_format_errors <= self.n_consecutive_format_errors:
                self.add_messages(
                    *e.messages,
                    {
                        "role": "exit",
                        "content": "RepeatedFormatError",
                        "extra": {"exit_status": "RepeatedFormatError", "submission": ""},
                    },
                )
            else:
                self.add_messages(*e.messages)
        except InterruptAgentFlow as e:
            self.add_messages(*e.messages)
        except Exception as e:
            self.handle_uncaught_exception(e)
            raise
        finally:
            self.save(self.config.output_path)
        if self.messages[-1].get("role") == "exit":
            break
    return self.messages[-1].get("extra", {})

run() has a while True loop that calls step(). step() calls query(), which passes the message history to the LLM and gets output back, usually a tool call, and that tool call gets executed.

2. Tools / environment:

def execute_actions(self, message: dict) -> list[dict]:
    """Execute actions in message, add observation messages, return them."""
    outputs = [self.env.execute(action) for action in message.get("extra", {}).get("actions", [])]
    return self.add_messages(*self.model.format_observation_messages(message, outputs, self.get_template_vars()))

execute_actions() takes the actions from the model's message and executes them based on whatever environment is configured: local subprocess, Docker exec, Singularity/Apptainer. Swapping environments only means swapping what self.env.execute() points to, nothing else in the code needs to change.

3. Context management:

def query(self) -> dict:
    """Query the model and return model messages. Override to add hooks."""
    if 0 < self.config.step_limit <= self.n_calls or 0 < self.config.cost_limit <= self.cost:
        raise LimitsExceeded(...)
    if 0 < self.config.wall_time_limit_seconds <= int(time.time() - self._start_time):
        raise TimeExceeded(...)
    self.n_calls += 1
    message = self.model.query(self.messages)
    self.cost += message.get("extra", {}).get("cost", 0.0)
    self.add_messages(message)
    return message

self.messages is the context passed to the model. At this level, mini-SWE-agent maintains the conversation history as the agent progresses; whether that constitutes the paper's stronger notion of active context management depends on how the model/history is actually selected, trimmed, or transformed during execution.

4. Control:

There's no dedicated method for this. It's enforced through the same if/else checks already visible in query() above:

if 0 < self.config.step_limit <= self.n_calls or 0 < self.config.cost_limit <= self.cost:
    raise LimitsExceeded(...)
if 0 < self.config.wall_time_limit_seconds <= int(time.time() - self._start_time):
    raise TimeExceeded(...)

Plus:

if self.messages[-1].get("role") == "exit":
    break

Together, these don't make the agent deterministic, but they make its execution bounded and give the harness deterministic stopping conditions: hitting a limit, timing out, or actually finishing the work.

By this definition, mini-SWE-agent satisfies all four conditions, at least on a loose reading of context management, so I'd classify it as a harness.

Is it a good harness?

Depends on the model underneath it. On the mini-swe-agent leaderboard, Claude Opus 4.5 (high) scores 76.80%, with Gemini 3 Flash (high) close behind at 75.80%. GPT-4o sits far lower, around 21.62%. The model clearly matters. But these numbers are also a reminder that we're not measuring the model in isolation, we're measuring the model running inside a specific agent system, since SWE-bench Verified uses mini-SWE-agent as the common harness for these evaluations. That makes the leaderboard useful for comparing model-plus-harness systems, but it doesn't cleanly separate how much of the performance comes from the model versus how much comes from the harness itself.

I tested it myself with a smaller model, Qwen3.5 4B, running locally on Ollama. It handled simple tasks fine. It failed on something slightly harder: building a Flappy Bird game. It failed on the physics, the bird crashed while passing through the poles, even though it should have made it through cleanly.

But I don't think the failure was purely a model problem. Part of it was the harness.

You could argue: if Gemini 3 does fine with the same harness, why doesn't Qwen3.5? Fair enough, it's a much larger model, obviously it'll do better. But we're not here to compare models. We're here to look at the harness. And a bash-only harness like this one is missing something specific: vision.

or more precisely, any real observation channel into what the application actually does once it's running.

Right now the harness can tell the model "the script ran without an error." It can't tell the model whether the bird actually cleared the pipe or crashed into it. That's the gap. If the model gets fed real evidence that its code is failing, not just confirmation that it executed, it can likely course-correct into something much better than one-shotting the physics blind. Something like Puppeteer or Playwright MCP would let the model test its own output, understand a screenshot, see where it actually failed, and fix it with that in hand.

Harnesses aren't limited to coding agents, either. They can wrap around anything an agent is capable of doing. But the harness itself has to shift a lot depending on the application. The right harness depends on where the model fails.

If the model loses track of long tasks, give it better context management. If it can't verify its own work, give it an evaluator. If it can't see the application's output, give it an observation channel. If it runs forever, give it control mechanisms.

The interesting part isn't building a bigger harness. It's figuring out exactly what the model needs help with, and engineering only that part around it.

What's next

I'm going to try building my own harness from scratch, leaning heavily on mini-swe-agent as a base since it's the simplest real harness out there. I'll also look at Pi and OpenHands for inspiration, though both are more specialized toward particular use cases.

Then I'll try the Flappy Bird task again, one-shot, with a harness, on the same small model (Qwen3.5 4B). My focus going forward is two things:

1. Getting better results out of smaller models through the harness itself.

2. Building a focused "friendly neighborhood harness," one that handles a specific problem and does that job properly, rather than trying to be general-purpose.

This is my first proper technical post, so a quick introduction: I'm a fresher AI engineer, and I'm using this blog to document what I actually build, test, and break while learning in public. I'm still figuring a lot of this out, so if I've misunderstood something, I'd genuinely like to hear about it. Follow along on X, and subscribe to the blog if that sounds useful. More of these coming.

References

• mini-swe-agent repo

• Harness Design for Long-Running Applications (Anthropic)

• Effective Harnesses for Long-Running Agents (Anthropic)

• What Makes a Harness a Harness (paper)