$ cat wiki/papers/2026/2610.35864-vibevideobench.md
VibeVideoBench: evaluating finished video delivery
TL;DR
VibeVideoBench evaluates agents that write code to render video and synthesize sound. The best evaluated system satisfies all core requirements on 76.3% of tasks, while counting undelivered videos as zero changes the ranking. (source)
Authors & Org
Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Han Li, Wentao Zhang, Weiran Xu, Kongming Liang and Zhanyu Ma. Affiliations are not established by the abstract page. First submitted 2026-10-26, revised 2026-10-27. (source)
Method
The benchmark has 239 English/Chinese tasks across 12 categories. Agents use their native harnesses in a sandbox without image/video generation models and must deliver an MP4 with sound within one hour. Evaluation combines delivery-specification checks, about 13k task-specific rubric checks, and five quality dimensions scored by three VLM judge panels. (source)
Results
Across nine systems and 2,151 scored rollouts, the authors rank GPT-6 Astra first and Claude Opus 5.5 second with both judges and human annotators. Three quarters of missing videos result from final renders starting too late, running too slowly or never starting. More frame inspection does not predict higher quality in their analysis. These are author-reported findings. (source)
Significance
For Agents (LLM Agents) and Eval Harness Configuration, producing a good intermediate artifact and delivering the final file are separate outcomes. The benchmark makes rendering-time allocation part of the tested system. (source)
Open Questions
How do results change with another time budget or rendering environment? The abstract does not resolve these questions or establish comparative total cost. Full paper, code and output videos were not reviewed. (source)
Cite
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation, arXiv:2610.35864, 2026. (source)