# Building Sadhan: My First AI Harness

After going through a lot of harness repos and research papers, I finally decided to build my own. I thought it would be simple, and the scaffolding part really was. A minimal working demo came together quickly. But the more time you spend on it, the trickier it gets.

In this post I'll cover where Sadhan is right now, the difficulties I hit, how I solved them, what I learned along the way, and what I plan to do next.

* * *

### Version 1: The Simple Loop

The first version of the harness was small. I gave it a bash tool that ran a single command, and the bash process closed after each one, so nothing persisted between commands.

The flow worked like this:

1.  The agent loop starts and initializes the state.
    
2.  The user's query is added to the state and passed to the LLM.
    
3.  The LLM returns a message with `<reasoning>` and `<action>` tags.
    
4.  A parser method pulls out the reasoning and action and returns them as JSON.
    
5.  The action part is executed as a bash command.
    
6.  The return code and output from the bash tool go back to the LLM, and the loop continues.
    

That is the whole idea. The model thinks, picks an action, the harness runs it, and the result goes back so the model can decide what to do next. This worked well for a while. I used it for single-file coding tasks and it handled them fine, so at this point I felt the hard part was behind me. It wasn't.

![](https://cdn.hashnode.com/uploads/covers/69ee1d2d3d6a492cdd0b6bbe/6d8b678f-acb8-4e4a-837f-c12048114460.png align="center")

* * *

### Problem 1: Bash With No Memory

Single-command bash breaks down quickly on real projects. Anything that depends on earlier commands, like working inside a Python venv or moving through nested directories, became really difficult without a persistent shell.

Think about a normal workflow. You `cd` into a folder, activate a virtual environment, and then run your tests. Each of those steps depends on the one before it. When every command runs in a fresh shell, the agent keeps losing its place, and the model has to work around a limitation that a human developer never has to think about.

So I replaced it with an asyncio-based persistent shell, where one shell process stays alive for the whole session and every command runs inside it.

### Problem 2: How Do I Know a Command Has Finished?

A persistent shell created a new problem. Before, a command ended when its process ended, so knowing it was done was free. Now the shell never exits. When the LLM sends a command, how do I know it has completely finished and isn't still running? Commands like `pip install` make this obvious, since they produce output over a long stretch of time. If I return too early, the model sees half the output and makes decisions on incomplete information.

My fix was a sentinel marker. After every LLM-given command, I append an echo of a unique marker plus the exit code:

python

```python
marker = f"__SADHAN_DONE_{token}_"
payload = f"{command}\necho {marker}$?\n"
```

The shell runs the command, and then prints the marker followed by the exit code of that command. I read the output stream until the marker shows up. When it does, I know the command is fully done, and I get the return code from the same line.

This change gave the agent much better power. It can now move around nested directories and work across multi-file projects, because the shell keeps its state from one command to the next.

![](https://cdn.hashnode.com/uploads/covers/69ee1d2d3d6a492cdd0b6bbe/fbe12bd8-f5bd-4196-9879-5eb1715675c4.png align="center")

* * *

### Problem 3: Tag-Based Output and Small Models

The tag-based LLM output gave me a lot of headaches. Small models like Qwen 3.5 9B would often write the opening `<action>` tag but never the closing `</action>`. That caused a lot of parsing errors, and many times the model just gave me broken results instead of something the harness could run.

I could have kept making the parser more forgiving, but that felt like patching a symptom. So I switched to tool-based calling. I know small models are not great at tool calling, but I tried it as an experiment and it gave me better results. It still isn't always correct, but it is a lot better than the tag-based approach.

### Supporting More Models

Until this point I had only tried local models, and there was no model registry for anything else. I added LiteLLM as the provider layer so I can switch models without touching the core loop. It doesn't cover every model yet, but the popular ones like Gemini, OpenAI, and Anthropic are in.

### Sessions and Modes

I also added session management. Conversations are stored by directory path and time inside `.sadhan/sessions/`. This makes it easy to pick up a project I started last week without starting over, and sessions can be renamed for convenience.

* * *

### How Sadhan Works Today

1.  The user installs it in a venv with `pip install sadhan` and runs `sadhan`.
    
2.  A TUI opens, and the user types the task.
    
3.  The state's message array gets the system prompt first, then the user's task.
    
4.  A `while True` loop runs. The messages go to the LLM, which reasons over them and returns a tool call if needed.
    
5.  The tool call is executed in the persistent shell and the output goes back to the LLM.
    
6.  This repeats until the task is done.
    

**Stopping conditions:** the loop can't run forever. If the same kind of error happens 5 times in a row, the agent terminates automatically and kills all child processes, so nothing is left running in the background. It also terminates when the default step limit is exhausted.

![](https://cdn.hashnode.com/uploads/covers/69ee1d2d3d6a492cdd0b6bbe/7208e2b7-cd2c-407d-a371-4a69ea7ce11c.png align="center")

* * *

### What I Learned

**The loop is the easy part.** The first version was basically LLM, tool, result, repeat, and it came together fast. Almost every real problem showed up around the loop: shell state, knowing when a command is done, the model's output format, and how to stop safely.

**Persistence changes everything.** Moving from one-shot bash to a persistent shell wasn't a small upgrade. It made the agent usable on real projects, but it also brought a new class of problems, like detecting completion, that I never had to think about before.

**Keep the model layer swappable.** Using LiteLLM means the loop doesn't care which model is behind it. That matters when you are experimenting with local models and hosted ones side by side.

**Small usability features matter.** Sessions look minor, but being able to come back to a project a week later without starting over changes how the tool feels to use.

**An agent needs a way to stop.** Without the 5 consecutive errors rule and the step limit, a confused model can keep looping and leave processes running. Termination and cleanup need to be designed in from the start.

**Building it is not the same as knowing it works.** I've been testing by feel so far. That is why evaluation is the next big thing on my list.

* * *

### What's Next

*   **Evaluation.** I haven't properly evaluated the agent yet. In a couple of weeks I'll run it on 3 to 4 real GitHub repos to fix bugs or add features, and share the results.
    
*   **Plan mode.** Along with an evaluator mode that runs in the backend and isn't used directly by the user.
    
*   **Multimodal input.** Right now it only understands text. I want to make it handle images and other data later.
    

Sadhan is still a work in progress, and I'll keep sharing what I learn as it grows. More updates soon.