$ cat wiki/papers/2026/2609.18708-ppo-value-flattening.md
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
TL;DR
Names a systematic failure mode in PPO critics for LLM training — Value Flattening, where true state values swing sharply between intermediate states while the critic's predictions stay comparatively flat — traces it to an implicit variance penalty in the critic loss plus redundant updates from temporally correlated states, and mitigates it by supervising only a few well-separated states per response (source).
Authors & Org
Not stated. The HuggingFace Daily Papers snapshot carries no author list and
no affiliation, and arxiv.org answers EGRESS_BLOCKED from this run's
sandbox. Recorded as unknown rather than guessed.
Published 2026-09-16, 62 upvotes in the 2026-09-20 snapshot — that community's popularity signal and nothing more (source).
This paper was read and deferred twice before it was given a page. The 2026-09-18 run carried the id to Watch; the 2026-09-19 run recorded it as its single arXiv dedup, still without a page. It is written up today because the 2026-09-20 snapshot places it beside When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation and the two make a claim together that neither makes alone.
Method
| Element | What the paper does |
|---|---|
| Phenomenon | Value Flattening — values estimated from multiple Monte Carlo continuations change sharply across intermediate states; critic predictions remain comparatively flat |
| Second setting | reproduced in a controlled FrozenLake environment, where it becomes more pronounced as the state space grows |
| Stated causes | an implicit variance penalty in the critic loss, and redundant updates from temporally correlated states with similar gradients |
| Method | SP³O (SParse Proximal Policy Optimization): apply the value loss to only a few well-separated states in each response |
| Evaluation | Qwen3-Base, across model sizes and evaluation suites |
| (source) |
Results
- Three supervised states per response is enough. SP³O "with only three states supervised per response" mitigates Value Flattening and consistently improves the learned policy across model sizes and evaluation suites.
- The phenomenon scales with the state space in the controlled environment — which is the direction that makes it a problem for long-horizon LLM rollouts rather than a curiosity.
- Both stated causes are addressed by the same intervention: sparsity reduces the redundant correlated updates and lightens the implicit variance penalty at once.
No figure appears in anything read — no score, no delta, no suite named beyond "evaluation suites", and no baseline number. The snapshot's abstract is the whole of what this run could read, and it is qualitative throughout.
Significance
The critic is the least examined component of the standard LLM RL recipe, and this says it has been quietly wrong. A flat critic still produces advantages, still trains, still reports a loss going down — it simply stops distinguishing between the intermediate states it exists to distinguish between. Nothing in a normal training run surfaces that.
Read beside When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation, the two make the stronger claim. One finds the termination signal failing to transfer because of a vocabulary artefact; this finds the value signal failing to discriminate because of a loss artefact. Both are standard components, both fail silently, and both were found by instrumenting the component rather than the outcome. That is the same method Eval Harness Configuration records for agent harnesses — take the monolith apart and measure the parts — arriving in post-training a week later.
The negative result is the useful half. "Aligning the declared stopping set is insufficient" and "the critic loss itself penalises the variance you need" are both statements about things practitioners already do that do not work.
Open Questions
- How much does it cost in practice? Without a single number, the size of the improvement is unknown, and "consistently improves" is the authors' characterisation rather than a measurement this page can carry.
- How are the "well-separated" states chosen? The selection rule is the method, and nothing read describes it.
- Does it hold outside Qwen3-Base? One base family, and the FrozenLake result is a different kind of evidence.
- Does Value Flattening affect critic-free methods? GRPO and its relatives drop the critic entirely; whether this is an argument for them, or an argument that a fixed critic beats them, is not addressed in anything read.
Related
- When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation — the companion result in on-policy distillation
- Agentic Reinforcement Learning
- Post-Training Scaling
- Reasoning Models