$ cat wiki/papers/2026/2609.22947-rewardverse.md
RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
TL;DR
Video reward models are asked to compress "is this video good" into one scalar, and the paper names the failure that follows: scalar drift — the scoring scale collapses or shifts across prompts, so the same number means different things in different places and the reward is unusable for RL. RewardVerse inserts a dynamic rubric between the query and the scorer: generate the evaluation criteria first, then score against them. Training is two-stage (RGPO): warm up the scorer on self-evolving seed rubrics, then jointly optimise the rubric generator for query-adaptive criteria while keeping the scorer aligned to human ratings. State of the art on the 16-dimensional EvalVerse benchmark, pointwise and pairwise (source).
Authors & Org
Not stated — the HuggingFace Daily snapshot carries no author block for this
entry, arxiv.org answers EGRESS_BLOCKED from this run's sandbox, and no search
pass was spent on it. Recorded as unknown rather than guessed.
Method
The diagnosis is stated precisely enough to be checkable, which is unusual for a reward-modelling paper: existing video RMs "directly map complex, subjective video quality into a single score without explicit evaluation criteria", and the consequence is scalar drift — the scale collapsing or shifting across different prompts.
The fix is an intermediate representation:
| Conventional video RM | RewardVerse | |
|---|---|---|
| Input | query + video | query + video |
| Intermediate | none | generated rubric (dynamic, query-adaptive) |
| Output | scalar | rubric-guided score |
| The rubric is described as a stable semantic anchor: because the criteria are | ||
| written down per query, the scorer is asked a bounded question instead of an | ||
| unbounded one, and the scale has something to be anchored to. |
RGPO (Rubric-Guided Policy Optimization) trains the two halves in order:
- Warm-up — the scorer is trained against self-evolving seed rubrics. The rubrics bootstrap themselves rather than being authored.
- Joint optimisation — the rubric generator is optimised to produce query-adaptive criteria while the scorer is continuously realigned to human ratings.
The stated inspiration is professional human annotation engineering — a human annotation pipeline does not hand annotators a 1–10 slider and hope; it hands them a rubric.
Results
- State of the art on EvalVerse (16 dimensions), on both pointwise and pairwise evaluation, plus external datasets.
- Scalar drift is reported as mitigated. This is the claim the architecture exists to support, and the snapshot gives no number for it — no measure of cross-prompt scale stability is quoted, so the central result is stated qualitatively here.
- The reward signal is described as robust and interpretable for RL in video generation; interpretability here means the rubric is readable, which is a property of the representation rather than a measurement.
No figure in this entry is quotable as a score. Unusually for a paper page here, the snapshot carries a results claim with no table and no percentages — "state-of-the-art" and "mitigates scalar drift" are the whole of it. That is recorded as a gap in the capture, not as a criticism of the paper.
Significance
Agentic Reinforcement Learning and Post-Training Scaling both turn on the same dependency: RL is only as good as the reward, and the reward is usually either a verifier (cheap, exact, narrow) or a learned model (broad, fuzzy, drifting). Video generation has no verifier — there is no unit test for "cinematic" — so it is forced onto the learned side, and this paper is about what goes wrong there.
The mechanism generalises past video, and that is the reason to keep it. Scalar drift is not a video problem; it is what happens whenever one number is asked to span heterogeneous prompts. A rubric generated per query is a way of making the reward model's question narrower without making its domain narrower — which is the same trade Learning to Discover Interesting Mathematics makes by reducing "interesting" to a ratio, and the same one Coding Agents for Generalized Task and Motion Planning Problems gets for free from a simulator. Three papers captured on one day, three ways of manufacturing a verifier where none exists.
Where it is weaker than it reads: the rubric generator is itself trained, so the drift has been moved rather than eliminated — a rubric generator can drift across prompts exactly as a scorer can. The paper's answer is the joint optimisation against human ratings, which makes human ratings the anchor of last resort, and their coverage is not stated.
Open Questions
- How much was scalar drift reduced? No measurement is quoted. The paper's central claim is the one figure missing.
- Can the rubric generator drift? Nothing read addresses whether the generated criteria are stable across semantically similar prompts.
- Whose human ratings, how many, and on what scale? The joint stage is anchored to them and their provenance is not stated.
- Does the rubric help the generator or only the scorer? SOTA reward-model accuracy is not the same as better generated video, and no downstream generation result is reported in the snapshot.
- Authors and affiliation, above.
Cite
arXiv 2609.22947 — RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling, 2026-09-19. HuggingFace Daily Papers, 2026-09-28, 24 upvotes — a popularity signal from that community and not a quality or importance ranking (source).