Harness Training

Project Repository: https://github.com/workofart/harness-training

So I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench tasks. To align on the definitions, “harness” means the system (e.g. Claude Code, Codex, ChatGPT web interface etc…) wrapping around the model (e.g. GPT-5.5, Claude Opus 4.7 etc…) that interacts with a specific environment. The harness controls what the model sees, what tools the model can use, and how environment responses are fed back to the model etc…

My initial attempt was mostly treating the self-improvement loop as an experiment design problem. I wrote about the learnings in my previous blog post What 1,000+ Harness Experiments Taught Me About Self-Improving Agents.

Since then I’ve completely revamped the way I run experiments (hint: determinism). This allowed me to reduce the experiment time, to extract more signal per experiment and to build a general PyTorch-like framework for “training harnesses”. In this blog post, I’m going to talk about how this is done and why it matters in the context of recursive self-improvement.

Here are some results from evaluating the same trained harness + model X:

Training the Harness

For a precise definition, this problem is framed as an agent-guided discrete program search with terminal reward and persistent search memory.

Here’s a map of the recurring concepts and the PyTorch analogy that inspired the framework (introduced later):

Setup

My setup is pretty basic, mostly to control costs and reach a decent iteration speed of one experiment (30 - 40 tasks) within ~45 minutes:

• Rented 5090 GPUs in the cloud (1 - 4 depending on how many concurrent tasks I’m running). The LLM inference providers don’t guarantee determinism, which I will explain later.
• Qwen 3.6 35B A3B FP4 quantized model: The model weights occupy ~24 GB, while the KV cache (with radix/prefix caching disabled) uses ~4 GB to support up to 10 concurrent requests, a limit imposed by the deterministic kernel(discussed later). The remaining memory is consumed by activations and CUDA overhead.
• LLM inference configurations: seed 12345, temp 1.0, top-p 0.95, 32k context length, max token count 8k, reasoning token count 6k. These were mostly constrained by GPU memory and aiming decoding time of below 5 minutes for the worst-case max token count, and I didn’t want to split the model across GPUs
• SGLang inference engine: it supports deterministic batch inference, which I will explain later why it’s a prerequisite for training the harness (bootstrap script)

Next, I will dive into the two types of tasks where I trained my harness: SWE bench tasks and Terminal Bench tasks.

Baseline Harness

It’s important to mention how I define my baseline harness because this is kind of like how one would initialize a machine learning model during model training. And it is very important to initialize your harness properly because otherwise it’s going to be very hard for the harness training to find good learning signals (gradient). For example, I’ve seen that if the baseline harness is too thin, without the LLM response handling logic, most of the training will be spent on creating error handling for the LLM calls. The other extreme is when the baseline harness is overloaded with 10+ tool definitions, at that point there’s too much bias. Maybe certain models are trained with that, but it’s definitely not a generic starting point.

In my baseline harness, I defined two actions in total for tool use: Run for running any shell command and Submit for letting the harness know it’s ready to call the task verifier to check the results. By no means is this a good baseline, but it’s a clean yet learnable state.

Training on SWE Bench Tasks

Training Setup:

• Trained on 39 SWE-bench tasks (django/pylint/pytest/sphinx/sympy) ¹
• Capped at max 110 steps per task or 75 mins timeout per task
• Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model
• Agent used as src/trainer/estimator.py:AgenticEstimator for training the harness: GPT-5.5 high via Codex CLI
• Git branch for this experiment with all the candidate commits from training

Result: 8/39 -> 14/39 solved over 29 experiment runs: 8 promoted, 20 rejected, 1 baseline.

There were 21 tasks out of 39 tasks that were never solved in any of the 29 runs. Based on experiment artifacts, a third of them hit the 110 step-cap, which I enforced to speed up the training iterations. But there’s definitely a trade-off here where if I extend the step cap, we can collect more training signals, but each epoch will run longer. Or maybe there’s some model capability ceiling or my LLM inference resources were too limited².

Training on Terminal Bench Tasks

Training Setup:

• Trained on 38 terminal bench 2.0 tasks ³. The tasks were selected to remove cyber/cryptography tasks since certain models refuse to complete it. And it was narrowed down to a subset to maximize both variety/difficulty and minimize task completion time.
• capped at max 110 steps per task or 30 mins timeout per task
• Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model
• Agent used as src/trainer/estimator.py:AgenticEstimator for training the harness: GPT-5.5 high via Codex CLI
• Git branch for this experiment with all the candidate commits from training

Two-stage experiment runs show non-deterministic factors hinder the self-improving capability

You can see in the plot below that I reset the baseline and dropped all the promoted harness changes around the 26th experiment due to identifying some non-deterministic environment factors that polluted the experiment results. You can even see that the rejected candidates before the reset had a very narrow but steady variance.

After the non-determinism fix and restart, the harness was able to quickly find good changes and beat the previous best score in less than half of the experiment time.

Note on the tie-breaker promotion: the harness training framework allows users to define secondary reward to break primary loss function ties. In the above training run, the secondary reward is defined as the following, evaluated in order:

• % of tool calls that are valid on the first attempt
• the number of steps used to solve the task

Biggest jump (14ef7381, 17 to 22 solves): entire LLM output response budget burned on reasoning with no tool call emitted, which previously killed the trial. The harness change proposed was to disable thinking and inject the repair prompt below. The 5 converted solved tasks were all long-horizon (solved in 67-90 steps).

reasoning_runaway_repair_prompt = (
    "Your previous response reached the output length limit while still "
    "reasoning, before it emitted any tool call. Stop reasoning now and reply "
    "with a single, concise tool call and no other text."
)

Evaluation Results

Is training the harness actually worthwhile to do? In this section, I’m going to go through some of the evaluation runs by holding the trained harness fixed and swapping out the underlying task LLM model. Here are the evaluation configurations.

• Trained on Terminal Bench 2.0 38 tasks³ out of 89 tasks. Git branch with all the trained harness commits.
• Evaluated on Terminal Bench 2.0 all 89 tasks. It’s obvious that there is overlap between the training set and the evaluation set. 23 out of 38 tasks were solved by Qwen 3.6 35B A3B FP4 quantized model with the trained harness, and it didn’t include task-specific mechanisms to “cheat”. We will also show that evaluating the same Terminal Bench-trained harness against SWE-bench, there are transfer learning gains too.

Generalized and transfer learning capability

This shows that for a frozen harness that was trained once using one model (Qwen 3.6 35B A3B FP4 quantized model), it can lift up the capability of a wide variety of models.

It also shows that if we train a harness on one task set (SWE-Bench)¹, that same harness can improve task-solving capability towards a different task set (Terminal Bench)⁴.

New capability

There are a couple of mechanisms in the trained harness that contribute to better task-solving abilities

In this plot below, you can see:

1. “failed verification” converted to “solved” (+23) because the trained system prompt added

   “run output is scratch: editing a file inside a python snippet or a here-doc that only prints does NOT persist your fix”

   and

   “the on-disk state at submit is the only thing graded”

   This targets a baseline failure mode where a model wrote the code in some ephemeral output and submitted an unchanged disk state. Inside the 23 converted “solved” tasks, they used the new file tools (~300 of ~1200 calls). DeepSeek 3.2 is the biggest beneficiary (+7).

2. “ran out of time” converted to “solved” (+24) from the trained harness spending half or less of the baseline’s LLM latency. Specifically, GPT-5.5 headless-terminal task reduced from 1755s to 221s, DeepSeek compile-compcert reduced from 1538s to 165s. The trained harness mechanisms that contributed were:
   • read clipped to 250 lines to cap environment (stdout/stderr) bloat
   • shorter completions for several models (output tokens per call: GPT-5.5 −19%, MiniMax2.5 −22%, GPT-OSS 20B −11%)
   • if the “Run” command/tool call times out, prompt the LLM with a nudge “change your approach rather than rerunning”, which avoids repeated 300s stalls.
3. “no valid tool call repeatedly” converted to “solved” (+6), because in the trained harness, more file tools were introduced (added “read”, “write”, “replace” to the existing “read”, “submit”). More specifically, the “write” and “replace” that persist edits quote-safely via base64, which improved “missing tool call”⁵ recovery from 69% to 94%, despite a byte-identical repair prompt. For invalid calls ⁶, the trained harness repair echoes the failed call back as something like “You sent: {calls}”, which improves a ValidationError⁶ recovery from 51% to 83%.

If we combine the above figure with the below figure, the “hit_timeout” to “solved” conversion would organically cause high step counts, since the same tasks that previously ran out of wall-clock are now being solved, which is driven by efficiency. MiniMax 2.5 has near-zero parse failures in both the trained and baseline harnesses, yet still gains +7 solves beyond 50 steps. Five are timeout/verification flips, suggesting the gains come from “persisting fixes”⁷ and write/replace⁸, not parsing. For GPT-OSS 20B, the fatal step rate fell from 9.4% to 1.9%, allowing trajectories to run longer (attempted steps increased from 5.6k to 9.5k) and enabling some runs to convert late. Tool call failures per step remained essentially unchanged at ~31%. This means that failures survive in the trained harness, and the surviving steps bill 43% more output tokens. The other model family’s long-trajectory wins come from the “timeout” and “failed verification” to “solve” mechanisms. GPT-OSS 120B gains solves while its trajectories shrink. This again shows that each model really has a different failure model or task-solving strategy baked into the weights.

If we decompose “Task Solve” mathematically as \(\text{Submitted} \wedge \text{Passed Verification}\), the trained harness could only achieve its goal through those two lenses, and that’s what it did. “Submit better” comes from persisting fixes⁷, plus the trained harness’s longer, more careful trajectories filtering weak submissions out of the pool. For DeepSeek 3.2, pass rate also rose as failing submissions stopped reaching submit at all. Only MiniMax 2.5 netted solves from submitting more, at the cost of a lower pass rate.

Token efficiency

For the genuine token efficiency wins, since the trained harness introduced more file tools, it shifted the tool call mix toward cheap calls. In GPT-5.5’s trained harness evaluation, read averages 433 output tokens vs 1,013 for the untrained baseline harness evaluation, and 374 of ~2,200 calls moved to read/replace (tokens per call 1,422 decreased to 1,151).

There are essentially two camps here:

1. With GPT-OSS 20B as the representative, its context blowup is rescued by the trained harness survival mechanism, so the task can still be solved at longer steps, but they pay for it using more input tokens

    baseline   steps  ▓▓▓▓▓  (breaker trips ~step 5, low recovery — dies)
               input  ██                          context never gets big
   
    trained    steps  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  (recovers 94%, runs on)
               input  ████████████████████████████ each step re-reads more
   
    attempted steps 5618 grows to 9472   total prompt +110%
    input/solve +73% output/solve +18%   total output +43%
   

2. With GPT-OSS 120B as the representative, its trajectories shrank while solving more, which is shown by the cost savings on the output tokens side.

    baseline   steps  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓  (wanders longer before solving)
               input  ██████████████████
   
    trained    steps  ▓▓▓▓▓▓▓▓  (tighter path to submit)
               input  ███████                     
   
    attempted steps 5834 decreases to 4290   input/solve −55%   output/solve −35%
   

Tokens-per-solve falls because trained harness’s first-order effect was to increase solve, which increased the denominator, which can be seen for most cases in the below figure.

We see divergence (baked into the weights) for token efficiency as well. In terms of the cost of solving a task, the two GPT-OSS models from the same family sit at opposite extremes. 120B reduced calls per solve by 36% through fewer model calls, while output tokens per call remained nearly flat (+1%). By contrast, 20B increased calls per solve by 32%: longer survival task solves outweighed an 11% reduction in output tokens per call.

Note that “model call retry %” didn’t change much: 20B 31% to 27%, 120B 9% to 12%, everyone else shows less than 1% gap between the baseline and trained harnesses. So \(\frac{\text{model calls}}{\text{solve count}}\) are driven mostly by \(\frac{\text{trajectory length}}{\text{solve count}}\), not repair traffic.

MiMo 2.5 behaves the opposite from GPT-5.5. MiMo 2.5 is the only model whose calls got longer (+19% per call) while getting fewer (−8% calls/solve). This means the model does more deliberation per step at fewer steps, net +9% pricier per solve. GPT-5.5 did the reverse (−19% per call, +2% calls).

A rising useful-share can mean opposite things per model:

• MiniMax 2.5’s biggest jump, where token shares tripled (from 13% to 41%) but only on +2 solves, is driven by tokens spent on solved tasks exploding from 12M to 56M, where 36.5M of that is just two tasks (winning-avg-corewars 21M, fix-ocaml-gc 15.5M, the 242-step timeout to solved flip). Its failed-spend barely moved. Maybe this harness is not suitable for MiniMax 2.5 on those tasks?
• For GPT-OSS 120B, the “failed-spend” fell from 221M to 98M (−56%), it’s mostly because failure itself got cheaper (shorter trajectories before dying or timing out). This is because of the same trajectory-shrinking mechanism⁹ in the trained harness.
• For GPT-OSS 20B, it’s an interesting one. Keeping 20B alive poured an extra 192M tokens into tasks that still failed versus extra 16M onto solved tasks. When combined with the above analysis for this model, there’s definitely some capability issue on the model side that’s making it inefficient to rescue. Or if there was a better way to detect this early on, we can let the harness “give up early” when running with less capable models.

Note on interpreting the numbers: “Figure: Efficiency Levers” and “Figure: Useful Share” denominators mix failed-task spend into per-solve/share metrics, so every number inherits the composition effects we talked about earlier (which tasks die, time out, or reach submit). “Figure: Efficiency Levers”’s identity (per-solve = per-call * calls/solve) is exact, but neither panel isolates “the same task got cheaper”.

Determinism

In order to achieve the evaluation results above with the trained harness, we need to understand the core prerequisite of training a harness.

Remember that I initially ran a lot of experiments without considering this factor and used multiple trials to “gain statistical power”? Well, that’s not scalable for this project where I don’t have enough compute power to speed up the experiment cycles. So I changed the way I run experiments by guaranteeing determinism. What do I mean by determinism?

Given the same setup, configs and harness etc…, I expect the exact same logits returned from the task LLM, same output tokens, same reasoning tokens and executing the exact same tool calls against the environment, returning the exact same results from the environment and reaching the exact same conclusions. To motivate this path, let me share my initial attempt without determinism.

Motivation in the self-improvement loop context

Before I worked on guaranteeing determinism, I ran over 1000 experiments. During those runs, I saw a lot of experiment noise that caused tasks to flip from “solved” to “unsolved” even though the experiment treatment (harness) didn’t change the critical path for that particular task. I tracked every mechanism firing with a 1-1 mapped metric to prove this. As a result, I spent a lot of time in designing a “good” candidate promotion criteria.

This example below is the LLM responses being non-deterministic:

• task: swebench django-14534
• model: gpt-5.5
• input: identical 751-token prompt, identical tool-schema, fingerprint (1000b896960b), request key 0b4c96fa…3cad, sent 39 times

• output:

  run A (exp-20260614-221243, step 4): list_dir    {"path": "/testbed"}
  run B (exp-20260615-022510, step 4): search_text {"query": "class BoundWidget", "root": "/testbed"}
  

At the experiment trial-level, trials are not deterministic, so experiment outcome depended entirely on coin flips:

Ok, so we saw the initial experiment setup was non-deterministic, but why do we need determinism here?

Imagine this: if a harness change (proposed by the harness training loop) had a very small effect towards a very small number of tasks and behavior change, from a statistics perspective, many repeated trials are needed to determine whether the “solve” was noise or a true effect of the treatment. But even then, that only provided confidence intervals, not a guarantee of a “good” harness change.

So is non-determinism bad? Not necessarily. Another way to think about this problem is: determinism is useful for credit assignment but once our policy (harness) is frozen, non-determinism can help with real-world performance by avoiding getting stuck in a local optimum.

I came to this realization through a random thought I had one day: Why do Reinforcement Learning (RL) environments like https://github.com/openai/gym iterate through each experiment so fast and allow for stable learning signals from each iteration? I realized the key missing component was determinism.

If we can ensure determinism, then every experiment outcome can be completely traced back to harness change solely. In the RL framing, this is like making credit assignment cleaner.

N=1

You might have seen that standard agentic evaluations need pass@k over many seeds to fight noise. That’s because the evaluations need to reflect real-world use-cases, and non-determinism is present in nearly all of those use-cases.

But in our harness training case, our goal is to use determinism to find a good harness without the noise. Once we commit to a finalized harness (frozen), we can use it with non-deterministic environments and LLMs.

Now that we’ve motivated the determinism efforts, let’s talk about the two factors that we need to take into account: LLM inference determinism and environment determinism. I will go over each one.

Deterministic LLM Inference

When we talk about LLM inference determinism, we’re usually talking about the deterministic logits output for the same model serving settings (e.g. LLM checkpoint, inference server version etc…). Note that “temperature” and other sampling args like “top_k” affect how we sample the tokens from a given logits/softmax distribution, but they don’t affect how those logits are generated by the model in the first place.

The root cause of non-determinism

There is a great blog post Defeating Nondeterminism in LLM Inference¹¹ on this exact topic which I’ve learned a lot from, so I won’t go into details.

The reason why non-determinism is the default for many inference engines is because it’s often more efficient to avoid the special handling required to achieve determinism. Oftentimes it requires special kernel implementations for determinism guarantees.

LLM inference providers don’t guarantee determinism

Initially, I used different inference providers to train my harness for the task LLM. But after trying out many providers, I realized determinism was only best-effort, never guaranteed. There are a couple of reasons that I found:

• Multi-tenant inference. We’ve talked about how different batch shapes would cause the matrices to be multiplied in different order, which causes non-determinism. This is amplified by the fact that LLM providers need to batch many requests from many users together to save cost (fully utilize GPU memory).

• seed parameter for inference is not guaranteed. Some providers support it, but some don’t. But even if they do, they never guarantee it (for example).

  Determinism is best-effort and is not guaranteed across model versions

• Rate limits / infra issues can cause the task-solving trajectory to diverge or even crash the run due to timeouts

What I used instead

For LLM inference options, in a naive case, if you just call the model’s forward pass via PyTorch, you would get deterministic results, assuming you have set all the seed values correctly. But if you start to use these inference engines like vLLM or SGLang, they have a lot more optimizations done to improve the LLM inference throughput and memory footprint that break determinism.

I eventually decided to go with SGLang as the inference engine, as it’s the one that explicitly said deterministic inference is supported by a couple of attention backends.

I’ll leave out the experiment notes for tuning the inference engine since it’s too distant from this “harness training” project. The main takeaways are:

• Setting --enable-deterministic-inference and setting a fixed seed is the first thing that I did.
• mixed-length requests by themselves don’t break determinism
• client concurrency itself doesn’t break determinism, it’s more about how the requests are batched
• even with radix caching disabled, the max number of active request slots (Mamba-slot cap for Qwen3.6-35B-A3B-NVFP4 model’s State Space Model) still has a ceiling before non-determinism slips in. In this case, active request slots had to be <= 10 to guarantee determinism for my setup¹² (bootstrap script)

LLM Determinism costs throughput degradation

In the below plot, you can see that we can guarantee determinism up to 10 concurrent requests on this diagnostic box, but after that, we observe drift. This also means that we lose out on higher throughput.

Some throughput levers I tested that failed the determinism gate:

• torch.compile incompatible with batch-invariant ops
• bf16 mamba dtype improves throughput by +12.7% but breaks determinism
• radix cache fails mixed-prefix gate
• NGRAM spec decoding wildly nondeterministic

Deterministic environment

Another factor that we’ve taken for granted is that the environment the LLM is interacting with (input to the LLM) is deterministic, which doesn’t always hold. We can see this in the example below.

Setup: temp=1.0, top_p=1.0, seed=12345, sampling from the exact same deterministic logits from the deterministic LLM inference engine

Both runs made the same tool call at step 15. Everything before step 15 was completely identical on both the LLM and environment side.

e = Exists(Product.objects.all())
q = Q()
result = e & q
print("Exists & Q:", result)

But the environment returned different stdout back to the LLM, due to memory addresses being random.

Run A (env stdout): Exists & Q: (AND: <django.db.models.expressions.Exists object at 0x7ffffd51dd30>, (AND: ))
Run B (env stdout): Exists & Q: (AND: <django.db.models.expressions.Exists object at 0x7ffffd544d30>, (AND: ))

After that, the LLM output diverged starting from Step 16:

Run A (LLM output tokens): <emitted a valid structured tool call>
Run B (LLM output tokens): <emitted the intended tool call as text inside reasoning_content, so the harness rejected it>

Determinism needs to be an end-to-end property of the loop, not just an LLM policy property.

Doing this in practice is not easy. We can set some seed or turn off some verbose logging, but oftentimes since we don’t own the environment, it’s a game of whack-a-mole. As you can see, I’ve constrained many non-determinism factors at the source (environment), but there are still many that leak out that cannot be easily controlled or may affect task solving setup.

In the training framework (in the next section), before comparing a candidate against a baseline, the framework “certifies” each task by re-executing the baseline’s recorded action chain in a fresh environment and checks that every step reproduces the same output. Any tasks that fork are excluded because we want to only capture candidate effects caused by the candidate’s change, not environment noise.

Comparing pre/post-determinism

Holistically, we can see that with the same candidate promotion criteria, the candidate promotion rate increased

General Framework

As part of this project, I was able to create a general framework that allows any harness to self-improve to a state that is model-agnostic and task-environment-agnostic. I call this “Harness Training”. It’s a discrete program optimization behind a PyTorch-like API. An estimator edits the harness policy, the framework evaluates the candidate against the baseline, and a criterion decides whether the candidate becomes the next baseline.

criterion = StrictPareto()
optimizer = GreedyMonotonic()
trainer = Trainer(
    config_path="config/train_harness.yaml",
    estimator=AgenticEstimator(
        backend=CodexAgentBackend(trace_dir=Path("experiments/codex-traces"))
    ),
    criterion=criterion,
    optimizer=optimizer,
)

# The current codebase in core.py is the baseline harness before this loop runs
# Each iteration in trainer.epochs(...) will:
# - AgenticEstimator makes a change to the core.py as a "candidate" harness improvement
# - run the "candidate" change in an isolated worktree to produce experiment artifacts
# - score the candidate against the baseline on the same task panel into a "loss"
# - kicks off AgenticEstimator to do a diagnosis on what worked, what didn't work and records into learning.md
for loss in trainer.epochs(30):
    # Records the baseline-vs-candidate verdict
    loss.backward()
    # Optimizer either fast-forwards the candidate change (git commit) as a new baseline or rejects it (preserved as git ref)
    optimizer.step()

That’s it. The inner optimizer.step() persists the candidate harness change if it was proven to be “good”. The outer for-loop is like program search or hill climbing.

The inputs for a training run are:

• task environment (e.g. SWEBench, terminal bench tasks)
• deterministic LLM inference endpoint with a frozen LLM checkpoint
• loss function (criterion on what’s good and bad, e.g. more task solves is good, more LLM tokens consumed is bad)
• optimizer (how to read loss and decide when to persist the harness changes)

Output: full experiment traces of how the harness performed over time to solve more and more tasks, including the exact raw input and parsed tool calls and raw reasoning traces of the LLM calls and telemetry of the experiment. Example

Core Capabilities

# src/env/new_env.py
# 1. The task record: instruction + grading assets, plus two metadata fields the
#    harness reads directly — the agent's wall budget and the cache identity.
#    `replay_id` fingerprints every observation-affecting env input, so changing
#    the verifier invalidates stale cached steps on its own.
_ENV_FINGERPRINT = ...


@dataclass(frozen=True, slots=True)
class Task:
    instruction: str
    tests_dir: Path
    agent_timeout_sec: float | None = 600.0
    replay_id: str | None = _ENV_FINGERPRINT


# 2. The loader: read your dataset, hand back records + an env factory.
async def load_tasks(
    *, task_ids: Sequence[str], environment: EnvironmentConfig, verify_cache: bool
) -> TaskSet[Task]:
    root = Path("benchmarks/benchmark-name")
    tasks = {
        task_id: Task(
            instruction=(root / task_id / "instruction.md").read_text(),
            tests_dir=root / task_id / "tests",
        )
        for task_id in task_ids
    }
    return TaskSet(
        kind=environment.kind,
        tasks=tasks,
        env_factory=lambda task, rollout_dir: Env(
            task=task, artifacts_dir=rollout_dir, verify_timeout_sec=300.0
        ),
    )


# 3. The env: pick a container image, say how to grade. Everything else —
#    container lifecycle, provisioning, shell execution, teardown — is
#    inherited from DockerTaskEnv.
class Env(DockerTaskEnv[Task]):
    _task_workdir = "/work"

    def _build_solve_env(self, task: Task) -> DockerShellSession:
        return DockerShellSession(image="python:3.13-slim")

    async def verify(self) -> VerifyOutcome:
        await self._start()
        await self._solve_env.upload_dir(
            source_dir=self._task.tests_dir, target_dir="/tests"
        )
        result = await self._solve_env.run(
            command="python -m pytest -q /tests",
            cwd=self._task_workdir,
            timeout=self.verify_timeout_sec,
        )
        passed = result.exit_code == 0
        return VerifyOutcome(
            verdict=VerifyVerdict(completed=True, passed=passed, error=None),
            output=RawEnvOutput(
                exit_code=result.exit_code, stdout=result.stdout, stderr=result.stderr
            ),
            reward=1.0 if passed else 0.0,
            info={},
        )


# 4. Register it, or it is unreachable: add the kind to EnvironmentConfig.kind
#    in src/config.py and a branch to benchmark() in src/env/base.py.

# src/trainer/estimator.py
class AgenticEstimator(Estimator):
    """The default estimator: both decisions are delegated to an LLM agent.

    Three open-ended ingredients:
      1. SEARCH SPACE — anything the agent can edit in its worktree
      2. POLICY       — the LLM itself, steered by a prompt
      3. MEMORY       — a free-form memo the agent rewrites each epoch
    """

    def __init__(self, *, backend: AgentBackend) -> None:
        # The backend owns tool execution and confines the agent's tools
        # to the worktree it is handed.
        self.backend = backend

    def propose(
        self,
        *,
        repo_root: Path,
        tracker: RunStore,
        target: TrainingTargetConfig,
    ) -> None:
        """One agent turn leaves a candidate patch in the worktree.

        > measure baseline            # harness: 21/40 solved at commit abc123
        estimator.propose(...)        # THIS FUNCTION
        > capture + measure candidate # harness: diffs the worktree -> commit def456,
                                      #          re-runs all 40 tasks against it
        """
        # The agent cannot reach outside its worktree, so stage copies of
        # the shared evidence inside it first.
        staged = self._stage_reads(tracker, worktree=repo_root, experiment_id=None)

        self.backend.run_turn(
            prompt=f"Read program.md; you are the proposer. Diagnose the prior run "
            f"from {staged.root}, then make one bounded improvement to "
            f"{', '.join(target.patch_paths)}.",
            repo_root=repo_root,
        )

    def diagnose(
        self,
        result: ExperimentResult,
        *,
        repo_root: Path,
        tracker: RunStore,
        target: TrainingTargetConfig,
    ) -> None:
        """The agent authors the memo; the harness checks and publishes it.

        > candidate result 23 > 21 -> "promoted", run saved
        estimator.diagnose(...)       # THIS FUNCTION
        > next propose() reads memo   # the feedback loop closes
        """
        staged = self._stage_reads(
            tracker, worktree=repo_root, experiment_id=result.experiment_id
        )
        draft = staged.root / "learning.draft.md"
        self.backend.run_turn(
            prompt=f"Rewrite the learning memo from run {result.experiment_id}: "
            f"bottlenecks, what was tried, what to try next. Write it to {draft}.",
            repo_root=repo_root,
        )
        # The agent only ever writes a draft; the framework validates it and
        # publishes it across the worktree boundary.
        tracker.publish_learning(draft.read_text())

# src/trainer/loss.py
class TotalTaskSolves(Criterion):
    name = "total_task_solves"

    def _compare(
        self, comparison: Comparison
    ) -> tuple[str, tuple[SecondaryRewardComparison, ...]]:
        # Label the verdict for the run record; () = no secondary tiebreakers.
        delta = len(comparison.candidate_solved) - len(comparison.baseline_solved)
        if delta > 0:
            return "higher_total_task_solves", ()
        if delta < 0:
            return "lower_total_task_solves", ()
        return "no_total_task_solve_improvement", ()

    def _loss_value(self, report: Comparison) -> float:
        # Negative loss selects the candidate; optimizer.step() then promotes it.
        return float(len(report.baseline_solved) - len(report.candidate_solved))

# src/llm/custom_completion_backend.py
class CustomCompletionBackend(CompletionBackend):
    def __init__(self, *, config: LlmProviderConfig) -> None:
        self.config = config
        self._client = AsyncCustomClient(api_key=os.environ[config.api_key_env])

    @classmethod
    def complete_duration_bound_sec(cls, max_tokens: int) -> float:
        # Watchdog allowance for one healthy attempt; every provider returns the
        # same transport timeout it enforces, so bound and behavior cannot drift.
        return stream_stall_timeout_seconds(max_tokens)

    async def _complete(self, request: CompletionRequest) -> Completion:
        # Providers implement _complete, beneath the inherited complete()
        # ladder that classifies foreign exceptions into typed failures.
        # (Wrappers do the opposite: they override complete() and forward.)
        raw = await self._client.complete(
            model=self.config.model_name,
            messages=request.messages,
            tools=request.tools,
        )
        return Completion(
            content=raw.content,
            tool_calls=tuple(
                ToolCall(name=call.name, arguments=call.arguments or "{}")
                for call in raw.tool_calls or ()  # SDKs return None when no tools fire
            ),
            finish_reason=raw.finish_reason,
            usage=Usage(
                prompt_tokens=raw.usage.input_tokens,
                completion_tokens=raw.usage.output_tokens,
                reasoning_tokens=raw.usage.reasoning_tokens,
            ),
            reasoning_content=raw.reasoning_content,
            # Slim JSON-safe envelope written to the trace; never parsed by policy.
            response={"id": raw.id, "model": raw.model},
        )

    async def close(self) -> None:
        await self._client.close()

# Wire it in, in both places, or the subclass is unreachable: one entry in
# backend_class() in src/llm/backend.py, and the same provider value added to
# the LlmProviderConfig.provider literal in src/config.py.

Plugins

Mostly for speeding up training, see note below for prerequisites.

Note:

• llm_cache only works if the thing it’s caching is a pure function of its cache key. (e.g. LLM cache key = exact request payload), but purity of the function (fixed seed + sampling params = same completion) is the model/provider’s job, not the cache’s. If the LLM inference is not deterministic, whenever it hits its first cache miss, the whole suffix of the rollout diverges. In other words, cache can preserve determinism, but not create it.
• step_cache for environment steps requires the environment to be deterministic so that training can speed up CPU intensive operations
• verify_cache must be a pure function of the diff, network responses are frozen by replay scope.

Future Research Directions & Discussions

• Determinism and sparse rewards: What does determinism remove from sparse-reward optimization, and what problems remain?
• Overfitting to one seed: multi-seed determinism (K fixed seeds, promote only changes that help across a majority), which recovers variance estimates without giving up reproducibility. This also connects to regularization.
• Order-dependence: do all the candidate commits still contribute in the final harness? or did later changes make earlier changes obsolete? Would a different candidate promotion order converge to a different (or better) point? I think there are parallels to model initialization and regularization in deep learning training.
• Harness vs model ceiling: if a failure disappears when the same trained harness drives GPT-5.5, it was a model ceiling. But if it persists across models, it’s harness territory. How can we detect this? Is there a value in model-agnostic harnesses?
• Harness baseline/priors: Do mechanisms transfer as a “harness prior”? The SWE-bench and terminal-bench runs independently discovered near-identical mechanisms (reasoning-runaway repair, work-exists-but-never-graded fixes). That convergence suggests a task-environment-agnostic core. Can we merge the harness changes from separate runs? Can joint multi-dataset training beat single-dataset training plus transfer? Is there a small universal set of harness invariants that should just be baked into baselines (model initialization)?
• Harness training as data generation for weight training: Right now, we are always using a frozen LLM model to solve tasks, while we change the harness during training. Do the trajectories generated from the trained harness contribute better quality SFT/RL data than baseline trajectories? Can harness improvements be distilled into the model (e.g. reasoning-runaway harness repair becomes unnecessary if the trajectories are part of the RL dataset for the LLM post-training). What about compute? Should it be better spent on harness search or fine-tuning the model weights directly?
• Harness training scaling law: solve rate vs number of experiments vs task count

Appendix

Interesting observations during harness training

• Self-diagnosed reward hacking: in the learning.md (cross-epoch shared memory), it noted the promoted candidate as “HOLLOW, the borderline task django-11087’s change was incorrect but dodged the penalty by never submitting”, so the eventual task result was “timed out” instead of “verifier rejected”
• Harness learned to catch the model lying to itself: pytest exit codes piped through | tail always return 0, so the task-solving LLM interpreted the test suite as passing and submitted broken patches. The next turn harness change rewrote the pipeline with pipefail. Converted sphinx-10323 at 96 steps. The baseline trajectory had submitted a wrong patch at step 108 believing tests passed
• Invent new tools: Added a dedicated search tool because the model’s ad hoc search commands, submitted through the general run tool, were inefficient, unreliable, and sometimes timed out