Aspire: Can Models Self-Evolve from Vague Goals?

TL;DR

A benchmark that supplies only a natural-language capability goal — "become a better physicist" — and hides the evaluation tasks. The agent must interpret the goal, choose data and update methods, construct its own training and validation signals, and decide when to evaluate. Result: vague goals redirect search effort toward goal interpretation; agents complete training and harness-editing loops routinely, but weight-level gains stay sparse and unstable, and the strongest evolved harness remains below the engineered Qwen-Agent reference (source).

Authors & Org

Not published in anything read. The HuggingFace Daily Papers snapshot carries title, abstract and upvote count but no author list, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. HuggingFace Daily Papers, 2026-09-04, 172 upvotes; arXiv publication date 2026-08-31 (source).

Method

The complaint the benchmark is built on: existing work on LLM self-evolution begins with tasks and evaluation metrics specified by humans, which reduces self-evolution to optimising an explicit objective rather than deciding what and how to learn (source).

ASPIRE therefore provides only a capability goal in natural language, and the agent must:

• operationalise the goal by choosing data and update methods
• construct its own training and validation signals
• decide when to evaluate

It supports both model-weight and agent-harness evolution in one interactive environment, and scores the resulting systems on a hidden, expert-authored set of 520 items spanning six goals.

Results

(source)

Finding	As reported
Effect of vagueness	search effort is redirected toward goal interpretation
Loop completion	agents routinely complete training and harness-editing loops
Weight-level gains	sparse and unstable
Best evolved harness	below the engineered Qwen-Agent reference
Failure modes named	training on mismatched data; trusting narrow self-evaluations
Longer search	continued search and training can erase earlier improvements
The last row is the finding. Not that self-evolution plateaus — that more of
it can be worse than less of it, because the agent's own evaluation signal is
narrow enough to reward a direction that a hidden held-out set does not.

Significance

Aspire and HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? were published a day apart, do not cite each other, and reach the same conclusion about harness evolution from opposite starting points. HarnessDev gives the agent a concrete objective and downstream execution feedback; Aspire gives it a vague goal and hides the evaluation. Both report that the loop runs and that the gains are unstable and do not transfer, and both find generated infrastructure behind a human-engineered reference — HarnessDev on code and search, Aspire against Qwen-Agent. Recorded on Eval Harness Configuration as this wiki's pairing.

Against Agentic Reinforcement Learning's self-improvement cluster, the distinctive move is removing the metric rather than the reward. J-Zero: Unified Challenger–Solver–Judge Co-Evolution from Zero Data replaces the judge's scoring basis; Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement removes the teacher's supervision and finds the student barely notices. Aspire removes the specification of what success is, and the result — the agent builds a narrow evaluation and then optimises against it — is the failure mode those two papers' settings are constructed to avoid.

Open Questions

• How narrow is "narrow self-evaluation"? The failure is named and not measured. Nothing read reports how far the agent's self-constructed validation set diverges from the hidden 520-item set, which is the number that would turn this from a diagnosis into a metric
• Six goals, 520 items, expert-authored — no per-goal breakdown was carried, so whether "become a better physicist" and a coding goal fail the same way is unstated
• Qwen-Agent is the only reference named, and it is a single engineered baseline. As with HarnessDev, "below the reference" has no scale without knowing how far above the seed that reference sits
• Does erasure have a shape? "Continued search and training can erase earlier improvements" could be overfitting to the self-made validation set, catastrophic forgetting, or reward hacking of a signal the agent authored. The three have different remedies and nothing read distinguishes them

Cite

Aspire: Can Models Self-Evolve from Vague Goals? arXiv:2608.31111, 2026-08-31. Recorded from HuggingFace Daily Papers, 2026-09-04 (source).