$ cat wiki/papers/2026/2608.25593-jit-agent.md
JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593)
TL;DR
A model trained to write agent harnesses. JIT-Agent formalises the harness as a composable artefact under a fixed four-module protocol and synthesises a task-adapted harness on the fly for any off-the-shelf agentic LLM. It states the thesis this wiki has been assembling for a month as its opening sentence — "Agent capability is not determined by the model alone" — and adds the claim that harness quality is trainable, transferable and compounding, "orthogonal to model scaling" (source).
Authors & Org
Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Shuicheng Yan (arXiv:2608.25593). Submitted 2026-08-26; cs.CL, cs.LG. No affiliations are shown on the arXiv listing, so no organisation is attributed here.
Listed on HuggingFace Daily Papers, 2026-08-27, 17 upvotes — that community's popularity signal and nothing more (source).
Method
The harness is defined as memory management, planning strategy, action protocol, and tool/skill orchestration — and the paper's diagnosis is that designing it "remains manual, task-specific, and fundamentally unscalable".
JIT-Agent is trained to do three things:
- Customise a harness for the task at hand
- Repair harnesses for stable and reliable execution
- Self-evolve, by distilling performance signals from an expanding archive of prior harness configurations
The harness is treated as a machine-generatable artefact governed by a fixed four-module protocol — the fixed protocol being what makes generation tractable (source).
Results
All figures vendor-stated by the authors; no independent reproduction is reported.
| Backbone | Benchmark | Reported gain |
|---|---|---|
| DeepSeek-V4-Flash | DeepSearchQA | +9.1 — surpassing GPT-5.6 |
| DeepSeek-V4-Flash | OdysseyBench | +4.3 — surpassing GPT-5.6 |
| GLM-5.2 | (not specified) | up to +20.2 |
| Generated harnesses are reported **performance-competitive with mature agent | ||
| runtimes** including OpenCode and Claude Code, and to improve the | ||
| DeepSeek V4, Mimo-V2.5 and Qwen3.6 families across scales | ||
| (source). |
Which benchmark carries the +20.2 is not stated, and neither is the harness the baselines used.
Significance
It converts a measurement problem into a training target. Eval Harness Configuration exists because the harness moves reported scores by more than a model generation does — that page's founding episode is a 4.9× spread on one model. Every response to that so far has been a warning: report the harness, don't compare across harnesses. JIT-Agent responds by training a model to produce the harness, which makes the confound a capability.
That cuts against the page's own remedy. If the harness is generated per task by a second model, then "report the (model, harness) pair" no longer identifies anything reproducible — the harness is an output, not a configuration. No reporting convention in use anywhere this wiki tracks distinguishes a harness-evolved score from a fixed-harness one, a gap Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses (arXiv:2608.08466) raised and this makes sharper.
It is the fourth self-improving scaffold in four days, and the fifth mechanism around a frozen model. With Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552) (a harness), Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses (arXiv:2608.08466) (a harness), FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills (arXiv:2607.21596) (workflow↔skill) and Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses (arXiv:2608.24876) (memory), the weights stay frozen and the thing around them accumulates. JIT-Agent is the first to make the scaffold itself the trained artefact rather than the evolved one.
The cluster is less independent than it looks. Zhaochen Yu and Shuicheng Yan appear on both this paper and Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses (arXiv:2608.24876), submitted a day apart into the same snapshot. Recorded because four papers converging is evidence and two groups converging is less of it.
Open Questions
- What are the four modules? The protocol is the load-bearing constraint and the abstract does not enumerate it.
- What benchmark gives GLM-5.2 +20.2? The largest number in the paper is the least attributed.
- What was JIT-Agent trained on, and does it overlap the evaluation tasks? A harness generator that has seen a benchmark's tasks is doing something other than generalising.
- What does the harness generator cost? Synthesising and repairing a harness per task is inference spend that no reported figure nets out.
- Is "orthogonal to model scaling" measured? It is asserted; nothing read varies scale and harness independently to show the axes do not interact.
Cite
JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (2026). arXiv:2608.25593.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041)
- Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses (arXiv:2608.24876)
- Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552)