How I Built a Harness to Run a Company of Agents

A goal in, a shipped feature out. The agents, what each one produced, the runtime we built to run them, and where it broke. A small, honest experiment on a lab project of mine.

What this is

I built a loop of AI agents that takes a single goal and turns it into a shipped, tested feature in a lab project of mine. This is the engineering writeup: the architecture, what each agent produced, how the handoffs work, how verification happens, and the failure modes.

Scope up front, because it changes how you should read the rest: this ran at small scale, in-process: a Python loop on my machine, not a managed orchestrator, against a lab project I built. The customer feedback that kicked it off was seeded by me. It ran one full cycle. That's enough to show how the machine is wired and where the seams are. It is not enough to tell you how it behaves at production scale.

The loop, end to end

Two rules hold the whole thing together:

• Every agent emits a typed object, validated before the next stage sees it. No prose parsing between stages.

• Agents recommend; humans gate actions. Nothing with a side effect happens without a person reading the diff.

The harness we built

The interesting engineering here is not the agents. It is the harness they run inside, the runtime that is everything except the model. We built it as a small Python package, and its whole job is to make a model fill a contract, run cleanly, reach its tools, swap backends, and never act without a gate. A few parts do the work.

One runner, many backends. A single function, run_agent(role, backend, output_model, ...), is the entry point for every agent in the loop. It resolves the backend and dispatches:

• Claude runs on the Agent SDK with an in-process emit tool (an MCP server the runner builds on the fly) that captures the agent's structured output as a tool call and validates it. It runs with setting_sources=[], so it inherits nothing from my machine.

• Codex runs as codex exec in an isolated temp directory, is prompted for JSON, and the runner validates the result against the same model.

• A local model runs over HTTP with its output constrained to a JSON schema: response_format on an OpenAI-compatible server like LM Studio, format on Ollama. It also does vision, which is handy for cheap, private screenshot review.

I ran the judgment stage (CEO) on Claude and the structured stages (GTM, PM, Engineer) on Codex. The agent definitions are identical across backends; only the runner changes.

The contract is the seam. Every stage emits a typed object, never prose:

On Claude this rides the emit tool: the agent returns its result by calling the tool, and the runner captures and validates that call. On the CLI and local backends it is prompt-for-JSON plus the same validation on the way out. Same shape, different mechanism. The contract is what lets the model behind any stage be swapped without touching the stage.

In-process tools, with a fallback for backends that can't hold them. Agents read their inputs through MCP tool servers the runner builds in-process (a repo tool, a data tool). The CLI and local backends can't host an in-process MCP, so for them the runner inlines the tool's inputs straight into the prompt. An id()-keyed registry maps each server to its data. The same agent works whether the backend speaks MCP or not.

Hermetic and concurrency-safe. setting_sources=[] keeps inherited hooks, settings, and keys out of an agent. Request-scoped state is passed as per-call keyword arguments, never mutated on a shared agent instance, so two runs in flight don't clobber each other. Both of these were lessons, not foresight. See "what broke."

Recommend-only orchestration. The only component allowed to mutate anything is the orchestrator, and it is single-write, idempotent, and forbidden from crossing a human gate. Agents produce findings, specs, and code; the orchestrator stages them; a human approves anything with a consequence.

A governance layer, partly built. Above the runner sits a control plane meant to read every agent's manifest, evals, and trace across the fleet: a registry, an eval-runner, a trace-auditor, an approval-router. This is the part I am most honest about: the design is an eight-agent control plane; what runs today is a subset. The intent is that nothing ships ungreen and every action is logged. I have not built all of it.

The agents: who did what, and what each produced

• CEO (Claude) takes the goal, produces CeoCall (ship_it, why, success_signal)

• GTM / voice-of-customer (Codex) takes the feedback list, produces CustomerSignal (top_request, frequency, quote, why)

• PM (Codex) takes the signal + CeoCall, produces FeatureSpec (name, what, target_module, acceptance)

• Engineer (Codex) takes the FeatureSpec, produces a module + a pytest

The actual outputs from the run:

GTM clustered six seeded feedback messages into one signal:

CEO read that against the goal and made the call:

PM scoped a buildable, testable feature:

Engineer wrote the module and five tests for it (trimmed):

Verification: the agent's test caught the agent's bug

The Engineer wrote the function and the tests for it. Four passed, one failed.

The function clustered feedback by its first four words. The test the same agent wrote expected the first three. So "confusing filters on results" and "confusing filters on category" looked identical to the test and different to the code. The model produced both halves in one call with no sign it noticed they disagreed.

The test was trustworthy not because the agent wrote it, but because it runs deterministically, outside the model's judgment. That is the load-bearing distinction in the whole system: verification has to be independent of generation. A generator that is equally confident when it is right and when it is wrong cannot grade itself.

The human gate

The loop is recommend-only. Agents produce findings, a spec, code, and a pull request. Nothing merges without a human reading the diff.

This isn't ceremony. Before I enforced it, the PM agent, given room, stopped proposing the feature and started building it, stepping into the Engineer's job. Two agents writing toward the same artifact make conflicting implicit decisions. The rule that fixed it: gate actions, not insights. Any agent can recommend anything; no agent takes a consequential action on its own.

From a unit to a feature

The Engineer's function did nothing on its own. Nothing called it. To make it real I wrote the wiring: an endpoint, an adapter that maps real records into the function's input shape, and a UI card.

Then I logged in as a user and confirmed it. The "At a glance" card rendered in the dashboard, built from real content and feedback.

The generation took minutes. The wiring and the check were the afternoon. A generated unit is not a shipped feature; the integration is the real distance, and a human walks it.

What broke

None of the failures were the model. They were all in the seams around it:

• An inherited shell hook crashed the agent with "Command failed with exit code 1." That message points at the agent's own command, so I looked there first. The real cause was a hook from my own machine firing inside the agent's environment. Fix: run hermetic (setting_sources=[]) so the agent inherits no hooks, settings, or keys.

• A leftover placeholder key (sk-ant-your-...) was being passed explicitly and overrode session auth, returning "Invalid API key." That reads like an account or billing problem. It was a fake key winning over a real one. Fix: stop passing it, use session auth.

• SDK/CLI version drift reported a successful run as an error result. The run had already finished and its output was captured; only the wrapper's status code disagreed. Fix: trust the captured output, not the wrapper's status code.

• A coding agent (Codex) read a real repo file instead of the inputs I handed it, because it could see the working directory and reached for what was nearest. The output was wrong, so the agent looked broken, but it had done exactly what it could see. Fix: run it in an isolated temporary directory so it can only see its inputs.

• Loading a 12B and a 26B local model at the same time froze the machine. It looked like a crash at first. Both models were resident at once, and their combined footprint went past the memory the machine had, so it was exhaustion, not a hang. It stopped responding entirely and the only way back was a force restart. Fix: run one large local model at a time.

At any real scale, the seam is the system. The model is the part that mostly works.

Scope and limits

• It runs in-process, a loop I start myself. I'm considering an orchestration layer (Paperclip) for a visible board and a heartbeat, but it would not change the gate.

• The "eval" here was a single unit test, not a suite. It demonstrated the pattern (independent verification catches the regression) at n=1.

• The feedback was seeded; there are no real users. This validates the machinery, not product-market fit.

• One cycle surfaces architectural failure modes (seams, self-grading, the wiring gap). It does not surface scale failure modes (concurrency, eval drift, distribution shift).

Takeaways

• Put the model behind a typed contract. It becomes a swappable dependency, and prose-parsing bugs disappear.

• Verification must be independent of generation. A deterministic check beats a model grading itself.

• Run the checks on every model change. That is what turns an upgrade into a diff instead of a leap of faith.

• Gate actions, not insights. One writer of record per artifact.

• A generated unit is not a feature. Budget for the wiring and the human who walks it.