$ cat wiki/papers/2026/2610.30932-userarena.md
UserArena: testing shopping simulators against production outcomes
TL;DR
UserArena evaluates interactive shopping simulators using real user trajectories and a controlled production A/B test. Its proposed OPTD training method most closely reproduces the policy effect among the evaluated simulators, according to the authors. (source)
Authors & Org
Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne and Zhaochun Ren. Affiliations are not established by the captured abstract page. First submitted 2026-10-22. (source)
Method
The benchmark pairs 64,729 session trajectories from 11,010 users with an interactive environment spanning 566,102 items and 13 action types. Evaluation covers output validity, local decisions, sequential behavior, personalization and outcomes, with offline validation against the production experiment. On-Policy Trajectory Distillation (OPTD) uses trajectory-level intent to supply dense teacher targets on student-generated rollouts. (source)
Results
The authors report that OPTD has the strongest overall simulator fidelity in the tested protocol and best reproduces the production policy effect offline. The abstract does not quantify that effect or provide uncertainty intervals; the dataset counts are not performance scores. (source)
Significance
For Agents (LLM Agents) and Eval Harness Configuration, this supplies a concrete external check on simulated behavior: agreement with outcomes from real interactions. That evidence applies to the reported shopping setting, not automatically to other domains or policies. (source)
Open Questions
Does simulator fidelity persist after repeated policy optimization or under a changed user population? The captured abstract does not establish either. Full paper, data and code were not reviewed. (source)
Cite
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents, arXiv:2610.30932, 2026. (snapshot)