FDE-Gym

TL;DR

FDE-Gym evaluates delivery when agents must discover unstated customer constraints and improve a business metric. Strong performance on specified tasks does not establish success in this setting. (source)

Authors & Org

Dongxu Yang, Hongda Yan, Renxin Wang and colleagues; the captured abstract page lists the complete author roster but does not establish affiliations. Submitted 2026-10-15. (source)

Method

The benchmark contains 95 tasks, adapted from 20 customer scenarios across 18 sectors, with scripted stakeholders and modifiable simulated systems. Scores normalize customer-metric improvement between 0 for the unchanged system and 1 for the reference solution, with interaction penalties. (source)

Results

• Among 13 model configurations from seven providers, the best realistic-setting configuration averages 0.56 and resolves 14 of 95 tasks. Resolution requires a score of at least 0.8 with no incident. (source)
• Even among 23 tasks where that configuration obtains every key stakeholder fact, only 10 reach normalized business uplift of 0.8 or above. Finding information alone does not close the delivery gap. (source)

Significance

For Agents (LLM Agents) and Eval Harness Configuration, requirements discovery and successful execution need separate measurement. The reported score is a benchmark result in simulated customer environments, not measured business impact in production. (source)

Open Questions

The abstract does not name the strongest configuration or give its total execution cost. Full-paper methodology, scripted-stakeholder realism and transfer to live customers remain unreviewed here. (source)

Cite

arXiv:2610.22350, first submitted 2026-10-15. (source)