$ cat wiki/papers/2026/2608.17426-semcomp-bench.md

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation (arXiv:2608.17426)

paperupdated 2026-08-23created 2026-08-23

TL;DR

Reformulates video generation as an outcome-oriented task: success requires the intended outcome to be achieved and semantically grounded in the reference image — and explicitly does not require a complete sequence of intermediate steps or conventional appearance consistency. SemComp-Data covers six domains; SemComp-Bench scores with a VLM answering structured binary questions, reporting an OA Score (Outcome Achievement) and a GR Score (Generation Reliability). Representative models find the combination hard (source).

Authors & Org

Not obtainable. arxiv.org is EGRESS_BLOCKED; the paper was not read. Listed on HuggingFace Daily Papers, 2026-08-23, 155 upvotes — the second-highest count in that day's snapshot. Upvotes are that community's popularity signal and nothing more (source).

Method

The reformulation. Semantic Task Completion Video Generation: a model is given a reference image and an instruction, and is judged on the generated outcome. Two things are required — outcome achievement, and semantic grounding, i.e. correspondence between reference image and generated outcome in task-relevant high-level semantics.

Two things are deliberately not required: presentation of the complete sequence of intermediate task steps, and conventional appearance consistency with the reference image.

SemComp-Data: an evaluation dataset over six domains. Each instance is a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip, produced by a four-stage curation pipeline that converts raw video into standardized instances.

SemComp-Bench: an evaluation protocol using a vision-language model to answer structured binary questions, reporting OA Score and GR Score.

Results

Stated only qualitatively: experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding remains challenging.

What the abstract does not give: any number at all — no OA or GR scores, no model names, no dataset size, and no agreement study between the VLM judge and humans.

Significance

The interesting move is what it drops. Appearance consistency with the reference image has been a standard video-generation metric; here it is discarded as orthogonal to whether the task was completed, and semantic grounding is defined at the level of task-relevant meaning instead. That is the same correction SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation (arXiv:2608.18701) makes for manipulation — score the outcome that matters, not the proxy that is easy to compute — arriving in a different modality on the same day.

The instrument is the weakness. SemComp-Bench scores with a VLM answering binary questions, which is the design FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents (arXiv:2608.18423) was built to avoid by using a deterministic scorer, and which SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation (arXiv:2608.18565) replaced with executed traces from a live runtime. A benchmark whose measurement is another model's judgement inherits that model's failures, and this abstract reports no agreement study.

Held at low confidence pending numbers. No scores reached this capture, so the page records a reformulation and a qualitative claim, not a result.

Open Questions

Any numbers — OA and GR scores for named models.
How reliable is the VLM judge? Binary questions are more robust than free scoring, but no human-agreement figure is given.
Does dropping appearance consistency change model rankings relative to existing video benchmarks? That comparison would establish the reformulation is doing work.
Author list, affiliation, dataset licence — unknown; the paper was not read.

Cite

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation (2026).
arXiv:2608.17426.

Referenced by

Eval Harness Configuration