$ cat wiki/papers/2026/2608.19741-thinkingbox.md
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741)
TL;DR
A sandbox with isolated MCP-compatible tool sessions, complete execution traces and outcome evaluation over terminal backend state, plus Thinkingbox-bench: 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT and consulting IT/HR support. The headline is the gap between two ways of counting the same runs: the strongest model reaches 65.36% pass@1 and 25.25% pass^20 — succeeding once is not succeeding reliably (source).
Authors & Org
No author list obtainable. arxiv.org is EGRESS_BLOCKED from this run's
sandbox and the paper was not read. The abstract gives the repository as
github.com/microsoft/thinkingbox, which attributes the work to
Microsoft
(source).
Listed on HuggingFace Daily Papers, 2026-08-26, 9 upvotes — that community's popularity signal and nothing more (source).
Method
The gap it names. Executable agent benchmarks already exist — code repair, web navigation, app APIs, function calling. What they miss is that consequential non-code work requires an agent to gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realise the correct persistent state transition without collateral effects. The last clause is the one the benchmark is built around.
The sandbox. Isolated MCP-compatible tool sessions, complete execution traces, and evaluation of the terminal backend state rather than of the response.
The benchmark. 507 policy-conditioned workflows across five named domains. Each attempt is scored by task-specific executable checks that "accept valid trajectories while rejecting wrong, missing, or extra effects"; designated tasks additionally check required properties of the final response.
pass^20 — success on all of 20 attempts — is the reliability measure, against pass@1, success on at least one.
Results
| Measure | Strongest model tested |
|---|---|
| pass@1 | 65.36% |
| pass^20 | 25.25% |
| Across proprietary and open-weight models. No model is named for either | |
| figure. |
The secondary finding is the sharper one: many failed trials terminate cleanly and take valid state-changing actions. So neither a clean termination nor a well-formed tool call predicts task completion — "response or tool-call-level signals are not clear proxies for end-to-end task completion".
Significance
It separates two things the agent literature routinely conflates, and the separation costs 40 points. A 65% pass@1 reads as a capable agent; a 25% pass^20 reads as one that cannot be deployed unattended on the same tasks. Both describe the same runs.
"No collateral effects" is the requirement that makes it a business benchmark. Rejecting extra effects — not just wrong or missing ones — is what separates an agent that completed the task from one that completed it and also did something else. Nothing else in this wiki's agent benchmark set scores that.
It is a direct counterweight to the day's lead paper. Apodex 1.1: Scaling Agentic Intelligence for Complex Work (arXiv:2608.23283) proposes "working capability — sustained, verifiable progress toward a real-world objective" as the unit of measurement and reports its own systems in "the leading performance band" with no number. Thinkingbox measures approximately the same construct, publishes both numbers, and the honest one is 25%. The two papers appear in the same snapshot.
It is the second MCP-native evaluation this wiki holds, and it treats MCP as the substrate rather than the subject — see MCP — Model Context Protocol.
Open Questions
- Which model scored 65.36 / 25.25? "The strongest" across proprietary and open-weight is unattributed, so nothing here can be joined to a model page.
- What is the spread? One model's pair of numbers says nothing about whether the pass@1 → pass^20 collapse is uniform across the field or specific to that system.
- Is the collapse variance or capability? pass^20 falls if the agent is unreliable or if a minority of the 507 tasks are effectively impossible; nothing read separates those.
- What harness? Given Eval Harness Configuration, a benchmark reporting a single number per model is a claim about a (model, harness) pair, and the harness is not described.
- Do the executable checks over-reject? "Rejecting extra effects" is the right requirement and also the easiest one to make too strict; no false-positive analysis is mentioned.
Cite
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents
in Stateful Business Workflows (2026). arXiv:2608.19741.