What an AI Harness Is

When developers compare AI coding tools they argue about models. Which one writes better C#, which one hallucinates less, which one is cheaper per million tokens. Then two products built on the same model behave nothing alike: one edits your repository and runs your tests, the other prints a code block and waits. The model is identical. The difference is the harness, and the harness is where most of the engineering lives.

The Term A model turns text into text, and nothing else

Strip away the product and a large language model is one function: context in, next piece of text out. It has no file system, no terminal, no network, no memory between calls, no idea how much it costs. It cannot open a file, run a build, call an API or remember what it did five minutes ago. Everything it appears to do, something else does for it.

That something else is the harness: the runtime that wraps the model, hands it tools, feeds it the results, keeps the history, and decides when to stop. If the model is the engine, the harness is the chassis, the brakes, the gearbox and the dashboard. An engine on a bench makes noise. A car goes somewhere.

The word comes from testing. A test harness is the fixture that runs code under controlled conditions and records what happened. An AI harness does the same job for a model, in production instead of in a test run.

The Loop Think, call a tool, check the result

An agent harness is a loop. It sends the current context to the model and asks for the next step. The model answers with text or with a request to call a tool: read this file, run this command, fetch this URL. The harness executes the call, appends the result to the context, and asks again. It stops when the model says it is done or when a guardrail says enough.

// The whole agent, minus the details. The model never leaves this loop on its own.
public async Task<string> RunAsync(string task, CancellationToken ct)
{
    var context = new Context(systemPrompt, task);
    var budget = new Budget(maxTokens: 200_000, maxToolCalls: 50);

    while (true)
    {
        var step = await model.NextStepAsync(context, ct);        // text in, text out, nothing else
        context.Add(step);

        if (step.ToolCall is null)
            return step.Text;                                      // the model is done

        if (!permissions.Allow(step.ToolCall))
            context.Add(await AskDeveloperAsync(step.ToolCall, ct)); // guardrail: a human decides
        else
            context.Add(await tools.RunAsync(step.ToolCall, ct));   // the harness touches the world

        budget.Charge(step);                                       // guardrail: stop before the bill does
        if (context.IsNearLimit) context = context.Compact();      // state: keep the history usable
    }
}

    

Every line that is not the call to the model is the harness. Tool execution, permission checks, budgets, compaction of a context that has grown past what the model can hold. None of it is machine learning. All of it is ordinary software engineering, and it decides whether the product works.

The Guardrails What the model cannot be trusted with

A model will happily propose rm -rf on the wrong directory, loop forever on a failing test, or spend a month of tokens in an afternoon. It is not malicious. It has no notion of consequence, because consequences happen outside the text. The harness owns them.

This is why two products on the same model feel so different. One harness lets the model run the tests and read the failure, the other does not. One asks before it pushes to a branch, the other never pushes at all. Same engine, different car.

The Other Harness Measuring instead of running

The same word covers a second thing. An evaluation harness does not ship features, it grades models. It serves a fixed set of tasks, hundreds or thousands of them, runs each model through the same agent loop with the same tools, and records what came back: pass rate, tokens spent, wall time. SWE-bench, where the task is a real GitHub issue and the pass criterion is the project's own tests, is the best known one for coding.

The reason it needs to be a harness and not a script is fairness. The tasks, the tools and the budget have to be identical for every model, the model must not see the answers, and the run has to be repeatable. When a vendor publishes a benchmark number, the harness is what makes that number mean anything, and a different harness will produce a different number for the same model.

The Rule Compare harnesses, not models

For a .NET team this is good news. Building a harness is building software: a loop, a tool interface, a permission model, budgets, logging. Swap the model behind it when a better one ships. The part that knows your repository, your tests and your rules stays, and that part is yours.