Learning to Solve Hard Problems in RL for LLMs by Never Giving Up

TL;DR

RL post-training makes models better at what they were already good at, and the paper names the mechanism rather than the symptom. Training LLMs with RL is shown not to improve performance equally across a dataset: large improvements on easy problems the model already solves, small improvements on hard ones. The authors call this the Matthew Effect in RL for LLMs, and argue the naive explanation — hard problems need more compute — is not the whole story: modern RL methods exacerbate it by wasting compute on easy problems. Their fix is Never Give Up (NGU), an adaptive sampling method that keeps generating samples for a problem until one is correct, which via asynchronous RL naturally spends fewer samples filtering out easy problems and more on hard ones (source).

Authors & Org

Not published in anything read. The HuggingFace snapshot carries no author list and no affiliation (source).

Method

Element	Detail
Diagnosed failure	the Matthew Effect in RL for LLMs — cumulative advantage, "the rich get richer", named after the phenomenon from economics and network science
Claimed cause	not only that hard problems need more compute, but that modern RL methods waste too much compute on easy problems and should dynamically reallocate it
Method	Never Give Up (NGU) — adaptive sampling that keeps drawing samples for a problem until one is correct
Mechanism	leverages asynchronous RL, so easy problems are filtered out with fewer samples and harder ones receive more compute
Design axis investigated	off-policy robustness, plus a stated set of best practices
The framing is the part worth keeping. The diagnosis is not "the model is weak on
hard problems" but "the sampler stops looking too early", which makes it a
claim about the training loop's compute allocation rather than about
capability.

Results

All figures are the paper's own; no independent replication appears in anything read (source).

Benchmark	Claim
Deepscaler (math)	NGU improves performance per compute, especially on harder problems
Manufactoria (a recent coding task)	standard GRPO with a per-test reward fails to fully solve problems that contain a range of easy and difficult tests; NGU iteratively improves, solving harder and harder tests, until it learns to fully solve them
No numbers are published in anything read — the abstract states directions
("improves", "fails to fully solve", "fully solve") and attaches **no score, no
delta, no compute budget and no baseline figure** to either benchmark. The
Manufactoria result is the stronger claim of the two, because it is a **qualitative
failure of GRPO** rather than a margin, and it is the one with nothing under it.

Significance

It is a compute-allocation argument landing in the same week as a compute-allocation measurement, from the opposite end of the pipeline. Yesterday this wiki recorded When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis, which measured where an agent's inference-time compute stops paying and named the crossover the scaling inflection point. NGU is the training-time version of the same question: where does the next sample stop being worth drawing, and modern RL answers it badly in the easy direction.

For Agentic Reinforcement Learning and Post-Training Scaling, the useful part is that the diagnosis is structural rather than about scale. "The rich get richer" says that adding RL compute uniformly widens the gap between what a model can already do and what it cannot — which is a direct argument that headline RL gains on aggregate benchmarks can rise while the hard tail does not move at all. This wiki has repeatedly recorded aggregate post-training gains with no per-difficulty breakdown published; this paper says that is precisely the breakdown that matters.

It is also a rebuttal of a convenient explanation. "Hard problems need more compute" is the reading that requires no change to anyone's method. The paper's claim is that the method is the problem, and — if it holds — that some of the reported difficulty of hard problems is an artefact of how samples were spent.

Open Questions

• How large is any of it? No figure, delta or budget is published in anything read for either benchmark. "Improves performance per compute" is unfalsifiable as stated.
• At what cost? Sampling until correct is unbounded by construction on a problem the model cannot solve. No cap, timeout, budget or failure mode is described in anything read, and that is the obvious place this method breaks.
• Does it survive contamination? Both benchmarks are named and neither is described; Manufactoria is called "a recent coding task" with no further detail.
• Is the Matthew Effect measured or asserted? The abstract states the uneven improvement as a demonstrated result but publishes no per-difficulty curve.
• Does it generalise past verifiable rewards? Both settings have a checkable correct answer, which is what "until one is correct" requires. Nothing read addresses tasks without one.

Cite

Learning to Solve Hard Problems in RL for LLMs by Never Giving Up.
arXiv:2609.13443, 2026-09-11.

• arXiv:2609.13443
• sources/papers-daily/hf-daily-2026-09-17.md — HuggingFace Daily Papers, 2026-09-17, 9 upvotes. Upvotes are that community's popularity signal and are not a ranking of importance; this paper was selected on interests.md weights, not on that count.