SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

TL;DR

Looped Transformers gain effective depth by iterating a shared block, but the literature compares them at fixed model size, which hands the looped model extra FLOPs and then credits the win to the architecture. SMELT re-runs the comparison with per-token FLOPs, non-embedding parameters and KV cache all matched, and the advantage survives: looping the middle half of layers twice saves 6.8–18.0% of training FLOPs on the compute-optimal frontier, fitted as a separate Chinchilla-style scaling law across four sizes up to 54B non-embedding parameters (source).

Authors & Org

Not published in anything read. The HuggingFace Daily Papers snapshot carries title, abstract and upvote count but no author list, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox, so no authors or affiliation are recorded rather than guessed. HuggingFace Daily Papers, 2026-09-03, 71 upvotes; arXiv publication date 2026-09-01 (source).

Method

The recipe is named in the acronym: Sparse MoE Transformer, middle layers Loop Twice.

Three budgets are held equal against an unlooped baseline (source):

Budget matched	Why it matters
per-token FLOPs	the one prior work let float
total non-embedding parameters	stops the win coming from capacity
KV cache	stops it coming from memory at inference
The arrangement was reached "through a series of ablations", and the paper then
fits a separate Chinchilla-style scaling law for each architecture rather
than comparing the two at single points — which is what lets the claim be about
the frontier rather than about one configuration.

Results

• 6.8–18.0% of training FLOPs saved on the compute-optimal frontier; SMELT's loss drops faster with compute
• Scaled across four sizes up to 54B non-embedding parameters
• The advantage transfers to downstream benchmarks beyond what validation loss predicts — largest on Code
• It grows with sample length and with the number of in-context examples
• Mechanistically: the second visit reduces the attention sink and redirects mass toward content-relevant tokens

(source)

No benchmark name or per-benchmark figure is given in the abstract, so "largest on Code" is recorded as the paper's word and nothing is quoted as a score. Nothing in sources/evals/ could check it in any case.

Significance

This is the rarer kind of scaling result: it does not claim a new frontier, it claims a previously reported frontier was measured against the wrong control — and then reports the effect anyway once the control is fixed. Both halves are needed. A paper that only announced the confound would be a critique; a paper that only reported the saving would be the thing it is criticising.

Two of the findings are more interesting than the headline percentage:

The gain exceeds what validation loss predicts. Compute-optimal scaling work routinely uses loss as the proxy for capability. Here the proxy undersells the architecture, and specifically on code. That is a data point against reading scaling laws straight through to downstream capability, in the direction people rarely check — see Post-Training Scaling and Test-Time Compute (Inference-Time Compute Scaling).

The gain grows with sample length and in-context examples. An architecture whose advantage is a function of context occupancy is not comparable at a single sequence length, which is how such comparisons are usually published.

The attention-sink account is a mechanistic hypothesis the paper offers, not a result it isolates — the abstract says the analysis "may underlie" the gains. Recorded with that hedge intact.

Open Questions

• 54B non-embedding parameters is the top of the fit. Frontier MoE models are an order of magnitude larger; whether the 6.8–18.0% band holds or narrows there is exactly what a scaling law is supposed to answer and exactly what four points up to 54B cannot.
• Looping the middle half twice is one recipe found by ablation. Nothing read says how sharp that optimum is.
• The result is on pre-training compute. Whether a looped block interacts well with the post-training and long-horizon agentic regimes that decide current model rankings is untouched.

Cite

arXiv 2609.01343, SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers, published 2026-09-01 — https://arxiv.org/abs/2609.01343 → (snapshot)