$ cat wiki/papers/2026/2609.22000-recreationworld.md

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

paperupdated 2026-09-22created 2026-09-22

agents computer-use coding-agents benchmarks evals

TL;DR

Computer-use and coding have been measured separately; this benchmark makes an agent do both, interleaved — explore a running application, implement a copy, run it and visually verify its own output. Five platforms, 250 held-out tasks. GPT-6 Astra leads at 58.1% overall and passes every programmatic test on 2.8% of tasks.

Authors & Org

Not stated. The HuggingFace Daily Papers snapshot carries the arXiv id, title, upvote count, publication date and abstract; it carries no author list and no affiliation, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. Recorded as unknown rather than guessed.

Method

The premise: computer-use agents have advanced along two separate lines — graphical interaction, and software development through code and the command line — while real digital work requires both, interleaved rather than stacked end to end (source).

The task is recreation: given a running reference, the agent must discover its behavior and build a faithful implementation, with no prescribed workflow. It decides for itself when to explore an interface, when to write software, and when to run and visually check what it built.

Reproducible environments on Ubuntu, macOS, Windows, Android and Web — five platforms — with a unified harness providing native GUI control and coding tools.
The running reference is the oracle for hidden behavioral tests, supplying execution-grounded rewards.
Trajectory generation is scaled against high-quality open-source applications.
RecreationBench, the held-out evaluation: 250 tasks across domains and platforms, with reference-grounded programmatic and visual assertions covering action-conditioned outcomes at multiple interaction depths. Each assertion is validated on the reference and by human reviewers before the suite is frozen for automatic scoring.

Results

Measure	Model	Result
Overall, RecreationBench	GPT-6 Astra	58.1%
Passes all programmatic tests	GPT-6 Astra	2.8% of tasks
The gap between those two rows is the paper's result. A 58.1% headline and a
2.8% complete-pass rate are the same system measured at two thresholds, and only
the second answers "did it actually rebuild the thing".

Three qualitative findings are reported without figures:

Agents reproduce static interface structure more reliably than interactions and computed outputs.
Generated applications remain smaller and more monolithic than their references.
Models trained on RecreationWorld trajectories improve across five out-of-distribution coding and hybrid computer-use benchmarks and more frequently verify their rendered outputs — offered as evidence of transfer beyond recreation. Neither the five benchmarks nor the gains are named.

The benchmark, environments and test suites are stated to be released; no licence or repository is named in the abstract.

Significance

This is the second paper in two days built on the same substitution, and the second to find that it costs the leaders a great deal. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks replaced the issue text with a working web application and watched a 100% → 64.0% collapse across restoration depth; RecreationWorld replaces it with a running application on five operating systems and reports 2.8% complete passes from the strongest agent it tested. Written up together with CodeMidas: Scaling Agentic Coding RL Environments from Code Itself, which does the same thing on the training side, the week's shape is that the specification channel is being taken apart the way the harness was in W37 — see Eval Harness Configuration.

The 2.8% figure is the one worth keeping, and it is a scoring-threshold result rather than a capability result. This wiki has repeatedly recorded agentic coding numbers with no stated denominator; here the paper publishes both thresholds itself, and they differ by a factor of twenty.

The GPT-6 Astra figure is also the third agentic-coding leaderboard in four days on which that model leads — after ProgramDistill's 49.2% and 2609.20804's harness sweep — and the first where a hybrid GUI-plus-code task is the axis. No comparison model is named in the abstract, so no margin over Claude Opus 5 or any other agent can be recorded, unlike ProgramDistill.

Open Questions

Which models were evaluated besides GPT-6 Astra. "Leads at 58.1%" implies a field and the abstract names none of it.
What 58.1% is the proportion of — assertions passed, tasks partially passed, or a weighted score. Without that, it is not comparable to the 2.8% beside it or to anything else.
The harness. A "unified harness with native GUI control and coding tools" is described but not specified, and this wiki records harness disclosure as the thing agentic scores keep missing.
Whether the trained models are released, and which base model they were trained from.
The five out-of-distribution benchmarks used to show transfer, and the gains on each.
Whether human reviewers validated the 250 tasks exhaustively or by sample.