$ cat wiki/papers/2026/2609.25804-taste-bench.md
The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks
TL;DR
Builds Taste-Bench, which measures not whether an agent finishes a long-horizon task but whether it chooses well at the forks along the way. The best frontier model answers 59.7% of the questions correctly, a larger reasoning budget does not help, and the capability can be distilled from a teacher that saw the outcome into a student that did not — improving end-to-end success on held-out SWE-bench Pro tasks (source).
Authors & Org
Not stated in the snapshot — no author list, no affiliation.
arxiv.org answers EGRESS_BLOCKED from this run's sandbox. Recorded as
unknown rather than guessed.
Method
The stated framing: on long-horizon tasks, the decisions an agent makes along the way — which hypothesis to test, which implementation to build on — determine the outcome of the whole run. The paper names the ability to make those decisions well the taste of an agent, and observes that existing benchmarks measure end-to-end success and none of them measures taste.
Construction, and this is the part that generalises:
- Each question presents a decision fork — a point in a real agent trajectory where multiple directions are available and one leads to a better outcome.
- The evaluated model chooses among the directions without seeing what happens after the fork.
- Forks are mined automatically, from parallel attempts at the same task and from detours inside a single trajectory, with no human annotation.
- The trajectories come from agents working real engineering and research tasks.
Training: the judgement of a teacher that has seen the outcome is distilled into a student. The student then makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.
Results
| Claim | Reported |
|---|---|
| Best frontier model on Taste-Bench | 59.7% correct |
| Forks whose deciding evidence appears later in the trajectory | much harder for every model |
| Effect of a larger reasoning budget | does not improve accuracy |
| Distillation from an outcome-aware teacher | better decisions on unseen tasks; improved end-to-end success on held-out SWE-bench Pro |
| Two of these are worth more than the headline. |
"A larger reasoning budget does not improve the accuracy" is a negative result against the prevailing direction of Test-Time Compute (Inference-Time Compute Scaling), which this wiki has tracked as the answer to hard-reasoning failures for most of a year. If taste does not respond to thinking longer, it is a different axis from the one test-time compute moves along, and that is the claim a reader should check first.
The late-evidence finding is a measurement artefact worth being careful about. A fork whose deciding evidence appears later in the trajectory is one where the correct answer is, by construction, not derivable from what the model was shown. Whether that measures poor taste or measures an unanswerable question is not addressed in anything read, and the paper's own framing ("much harder for every model") does not distinguish them.
No absolute numbers accompany the distillation result — "better decisions" and "improves end-to-end success" carry no figure in the snapshot, so the trained-taste claim is directional only.
Significance
This is an instrumentation result, and that is now four consecutive weeks of them leading the paper picks. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks moved the specification channel, Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents measured whether an agent knew it had succeeded, Harness-Zero: Harness Distillation via Agent-as-Harness and RRSI: Regularized Recursive Self-Improvement of Agent Harnesses took apart the harness. Taste-Bench takes apart the trajectory: the unit of measurement stops being the run and becomes the fork inside it.
The construction is the transferable part. Mining decision forks from parallel attempts at the same task requires no human annotation and no new task authoring — it reads evaluation data that agent benchmarks already generate and throw away. That places it in the same family as the four papers of 2026-09-18 to 2026-09-22 that removed the human-written task statement, and Eval Harness Configuration is where the family is tracked: a benchmark built out of another benchmark's discarded traces inherits that benchmark's harness, and no harness is named here either.
The distillation half points the other way, at Agentic Reinforcement Learning: an outcome-aware teacher supervising a student that cannot see outcomes is hindsight supervision, and the claim that it transfers to held-out SWE-bench Pro is the paper's only end-to-end evidence.
Open Questions
- Which model scores 59.7%, and what do the others score? "The best model" is the whole identification, and a benchmark's difficulty is not readable from one number.
- Is the late-evidence result a capability finding or an unanswerable-question finding?
- How much does distillation buy? No figure is given for either half.
- Does automatic fork mining select for forks where the better direction is recognisable? Forks are kept because one branch led somewhere better — which is a property of the trajectories that were run, not of the decision space.
- What harness produced the source trajectories? Unnamed, as usual.
Cite
arXiv 2609.25804, The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks, HuggingFace Daily Papers 2026-09-24, 111 upvotes (snapshot).