$ cat wiki/papers/2026/2608.20061-moe-hyperparameter-transfer.md
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts (arXiv:2608.20061)
TL;DR
Sweeping for the optimal learning rate is prohibitive at trillion-token, MoE scale, so this predicts it instead — in two transfers. First a Maximal Update Parameterization (μP) adaptation for MoE using Multi-head Latent Attention and the Muon optimizer, under which the optimal learning rate transfers consistently across model widths. Then a predictive scaling law along the token axis: linear regression on optima from small proxy models on limited budgets extrapolates the ideal learning rate to 10 trillion tokens at R² = 0.95. Validated by pretraining a 155B total / 17B active foundation model from scratch (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this run's sandbox; the
paper was not read. Listed on HuggingFace Daily Papers, 2026-08-25, 32 upvotes
(source). The abstract
says "our foundation model", so the authors are a lab that pretrains at 155B —
which one is unstated in the snapshot.
Method
The gap it names. MoE expands capacity without proportional compute, but hyperparameter optimization — "particularly the learning rate" — "at extreme scales of both model size and token budget via sweeping remains computationally prohibitive". The cost being attacked is the search, not the training.
Step one — transfer across width. A μP adaptation for MoE architectures built on Multi-head Latent Attention (MLA) and the Muon optimizer. Claim: under this parameterization, optimal learning rates transfer consistently across width-scaled models. This is the standard μP promise, made to hold for a specific modern MoE recipe rather than for dense transformers with Adam.
Step two — extrapolate along tokens. μP says nothing about the token budget, so a second mechanism is added: linear regression on the optimal values derived from small proxy models on limited budgets, producing a predictive scaling law that extrapolates to massive training horizons (e.g. 10 trillion tokens) at R² = 0.95.
The claim the two steps make together: proxy training on small models is sufficient to determine the optimal learning rate for full-scale MoE training.
Results
- Learning-rate transfer across width: reported as consistent; no figure given in the abstract.
- Token-axis extrapolation to 10T tokens at R² = 0.95.
- The validation is a real training run, not a benchmark: a 155B total / 17B active foundation model pretrained from scratch, with "stable training and evaluation results" offered as evidence the predicted configuration was right.
What is missing is the counterfactual. No swept-baseline comparison appears in the abstract — nothing says what the optimal learning rate actually was at 155B, or what a wrong one would have cost. "Stable training" is an outcome, not a margin.
Significance
It is a scaling-law result about the search rather than about the model, which
is the axis this wiki has been thin on. Every scaling claim currently on
Post-Training Scaling and in Test-Time Compute (Inference-Time Compute Scaling)'s
## Scaling Laws (open question) section is about where capability comes from
— parameters, post-training, or inference compute. This one is about what a lab
has to spend to find out, and it argues the answer is: a regression on small
proxies.
Read against the week's "death of params" claim, it is the quiet counterpart. Z.ai's Jie Tang argued on 2026-08-20 that parameter count is meaningful only alongside data, compute placement and deployment conditions — an argument about which axis to scale. This paper takes the axes as given and attacks the cost of configuring them, and it is the only claim of the two with a stated fit statistic.
The scope is narrower than the title. It is one hyperparameter — the learning rate — under one parameterization (μP), one attention design (MLA) and one optimizer (Muon). Whether the transfer survives a different optimizer, a different sparsity ratio, or depth scaling rather than width scaling is untested in anything read.
Open Questions
- Whose 155B model is this, and is it released? Unstated in the snapshot.
- What was the swept optimum? With no baseline sweep reported, the prediction has an R² but no error bar in loss.
- Does it transfer across depth and sparsity, not just width? μP's guarantee is a width guarantee; MoE has at least two more scaling axes (expert count, active fraction) and neither is mentioned.
- Is Muon load-bearing? The parameterization is derived for it; nothing says whether the result survives Adam.
- Does the token-axis regression hold past 10T? It is fitted on limited budgets and extrapolated; the abstract reports fit quality, not held-out validation at scale.
Cite
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for
Large-Scale Mixture-of-Experts (2026). arXiv:2608.20061.