Decoded
← Feed
arXiv:2608.23041AICLLG24 aug 2026

AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

1 min readHN ×4PDF ↗
AUTOSADDLER62%GEPA54.6%META-HARNESS53.2%DEFAULT AGENT53%0100% GAIA2 TEST PASS@1
fig. gaia2 test pass@1Auto-patched harness beats base and other optimizers

An AI coding agent debugs another agent's failures and patches its wrapper code, raising success rates.

9.0pp

Pass@1 gain on GAIA2

Similar gains on coding and terminal benchmarks, without manual harness tuning

How it works

  1. Harnesses are hand-tuned

    Agents fail on long tasks. The prompts, tools and control code around them are tuned slowly by hand.

  2. Debug traces, patch code

    An agent reads failed runs and the harness source. It writes typed patches and keeps those that help unseen tasks.

  3. Better agents, fewer runs

    Test success rose 9 to 10 points on three benchmarks. It beat GEPA and Meta-Harness using fewer task executions.

failed tracespatchif improvedlessonsnext iterationRunmini-batchDiagnoseand patchVerifyon batchDev-setcheckReflectinto EvoDAGEvolve nextharnessfailed tracespatchif improvedlessonsnext iterationRunmini-batchDiagnoseand patchVerifyon batchDev-setcheckReflectinto EvoDAGEvolve nextharness

What they found

Which comes out on top on GAIA2 test Pass@1?

Rollouts spent vs. dev accuracy

try it

Drag the rollout budget up and watch the best dev accuracy flatten.

Task executionsBest dev accuracyfew runs, weak harnessmany runs, plateauAutoSaddler peak near here
Best dev accuracy
medium
here
few runs, weak harness
Accuracy saturates early, so picking which runs to evaluate matters more than running more tasks.Illustrative model, not the paper's data

Terms

6 terms

Harness Everything around the LLM: system prompts, tools, and agent-loop control code.

Like a senior engineer reviewing crash logs from a batch of jobs, filing targeted fixes, and merging only those that pass staging tests.

Where it breaks
The engineer here is itself an LLM, and its diagnoses can be wrong.

Limits

  • Only one or two optimization runs per method, because runs are expensive.
  • Assumes stateless tasks, so memory and skill libraries are not optimized.
  • Costs more per patch ($14.56) than Meta-Harness or GEPA.
From the paper