AgenticBBO-Bench — agents under a finite optimization budget

TL;DR

AgenticBBO-Bench compares black-box optimization methods across five domains under a unified finite-budget protocol. The authors find that adding numerical tools does not consistently improve an agent. (source)

Authors & Org

Ming Chen, Rong-Xi Tan, Ke Xue, Yu-Jie Zhou, Taiye Lu, Zhi-Xuan Gao, Peng Xie, Zijun Shen, Chen Lu, Haopu Shang and Chao Qian. First submitted 2026-10-08. Affiliations are not stated in the captured abstract page. (source)

Method

The benchmark spans synthetic functions, hyperparameter optimization, database tuning, chip design and molecular design. It isolates optimization tools, task information and prior knowledge, and the LLM’s role during search. A separate five-task frontier challenge evaluates seven LLMs under the Codex agent harness. (source)

Results

• Agentic optimization achieves higher family-averaged scores than direct LLM methods in all five domains and outperforms the best numerical optimizers in four. These are domain-level comparisons, not a claim to win every task. (source)
• Task semantics are broadly useful; specific priors are less reliable. Numerical optimizers can absorb gains from trajectories initiated by an agent, while more numerical tools do not consistently help. (source)
• GPT-6 Astra and DeepSeek-V4.1-Flash lie on the performance-cost Pareto frontier among the seven tested LLMs on the frontier challenge. The abstract provides no score or price table. (source)

Significance

For Agents (LLM Agents) and Eval Harness Configuration, the useful intervention is holding the objective-evaluation budget explicit while varying the agent’s information and tools. Interpretation: a larger tool menu is a hypothesis to test, not an automatic capability gain. (source)

Open Questions

The abstract does not specify per-task budgets, repeated-run uncertainty, exact model settings or the complete cost accounting. Its Pareto result is limited to the evaluated challenge and configurations. Full paper and implementation were not reviewed. (source)

Cite

arXiv:2610.12183 · Code · (source)