$ cat wiki/papers/2026/2608.08888-full-bandwidth-transformer.md
Full-bandwidth transformer (arXiv:2608.08888)
TL;DR
Widens the vertical channel between decoding steps: instead of only the sampled token returning to the bottom of the stack, the previous top-layer hidden state is fused with the token embedding through a gated linear unit and fed back as the next input. Trained at 1B parameters to 400B tokens, it is reported to match or approach standard transformers trained on roughly 1.5× more tokens, with negligible per-token decoding overhead, while producing shorter reasoning traces at equal or better accuracy (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-17, 16 upvotes — that community's popularity signal and nothing more (source).
Method
The framing is a bandwidth asymmetry in the standard autoregressive transformer, stated in terms of two axes of computation (source):
| Axis | Channel width |
|---|---|
| Horizontal — across generated tokens | dense attention gives "broad horizontal access to the past" |
| Vertical — between decoding steps | "narrow": only the sampled token returns; the top-layer hidden state is discarded |
| Latent feedback is the proposed fix. At each decoding step the previous | |
| top-layer hidden state is **fused with the sampled token embedding through a | |
| gated linear unit** and fed back as the next input. The stated consequence is | |
| that "non-verbalized computation" can **re-enter the stack with a renewed depth | |
| budget** — the model gets to keep thinking about something it did not have to | |
| emit as a token. |
Three properties are claimed to be preserved: the standard transformer architecture, the KV cache, and the language-modeling objective.
Training it without losing parallel teacher forcing needs a scheduled multi-pass objective: latent feedback is introduced late in pretraining, and a small fraction of deeper feedback passes is mixed in for stability.
Results
1B-parameter models trained to 400B tokens (source):
| Measure | As reported |
|---|---|
| Validation loss | improved |
| 5-shot language-model evaluation | improved |
| Math and coding generation | improved |
| Instruction-tuned performance | improved |
| Equivalent standard transformer | trained with ~1.5× more tokens |
| Per-token decoding overhead | "negligible" |
| Reasoning traces | shorter, at equal or better accuracy |
| Every improvement above is directional — the abstract names no benchmark scores. |
What the abstract does not give: the benchmarks behind "5-shot language-model evaluation", any absolute number, the GLU's parameter cost, what "late in pretraining" means as a fraction, the size of the deeper-pass mixture, or whether the 1.5× data-efficiency claim holds at any scale other than 1B.
Significance
This wiki's Weekly Synthesis — W33 (2026-08-10 → 2026-08-16) gave last week a one-line spine: capability stopped arriving in the weights. Four releases explicitly trained nothing new, and four unrelated papers froze the model and evolved the harness around it.
This paper runs the other way, and that is why it is here. It is a change to the pretraining architecture whose entire claim is a data-efficiency ratio — the oldest form of scaling result there is, on the axis the week's other work argued had gone quiet. A single counter-example does not overturn the theme, and at 1B parameters it is well below any model on this wiki. But a theme that only ever accumulates confirming evidence is a theme nobody is testing, and the subtractive monthly digest will have to decide whether "capability left the weights" is still standing with this on the table.
The shorter reasoning traces at equal or better accuracy line connects it to Test-Time Compute (Inference-Time Compute Scaling) from an unusual direction. Nearly everything on that page moves compute around at inference time — Thought-Level Beam Search for Reasoning (arXiv:2608.08020), arriving in the same batch, reallocates it between traces. This paper claims to need less of it because computation that would have been spent re-deriving a thought through emitted tokens is instead carried forward in latent form. If that holds, the verbalised chain of thought is partly a transport mechanism, not only a reasoning one — and its length is an artefact of the vertical channel being one token wide.
Open Questions
- Does it scale past 1B? The whole claim is a data-efficiency ratio, and ratios of that kind are the ones that most often close with scale. Nothing read reports a second model size.
- What is "negligible" overhead, measured? A GLU fusion per decoding step is not free, and the claim competes directly against the 1.5× token saving.
- Is the KV cache genuinely unchanged? The abstract asserts preservation, but a feedback path from step n's top layer into step n+1's input is a sequential dependency; how it coexists with cached parallel decoding is not described.
- Does latent feedback survive interpretability? Non-verbalized computation re-entering the stack is precisely the state Mechanistic Interpretability reads through the emitted trace. A shorter trace at equal accuracy means more of the reasoning is no longer written down.
- Author list, affiliation, code availability — unknown; the paper was not read.
Cite
Full-bandwidth transformer (2026). arXiv:2608.08888.