$ cat wiki/papers/2026/2608.16157-freetoken.md
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution (arXiv:2608.16157)
TL;DR
An edge-native MoE serving system that treats a personal machine as a unified elastic inference platform rather than a small GPU, refusing to commit to a fixed offloading strategy and instead remapping computation and model state onto whatever resources are actually free. Reported reach: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on a single workstation GPU (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-20, 57 upvotes — that community's
popularity signal and nothing more. The system is stated to be released at
flashml.ai
(source).
Method
The stated premise is that serving frontier open-weight models still assumes datacenter infrastructure, and that two realities of local AI break the usual answer:
- Agent workloads continuously change their execution pattern — so a strategy tuned to one phase is wrong in the next.
- Edge hardware exposes heterogeneous resources whose balance differs per machine — so a strategy tuned to one machine is wrong on another.
The response is co-design across the whole serving stack rather than a single optimisation: model layout and loading, expert residency, CPU–GPU execution, agentic state reuse, and runtime memory management — all driven by bandwidth-adaptive execution rather than a fixed offloading plan.
Results
| Claim | Figure |
|---|---|
| MoE models supported | 20+ |
| Hardware range | 8GB laptop GPU → single workstation GPU |
| Laptop | 35B model |
| Gaming desktop | 284B model |
| Single workstation GPU | 753B — GLM-5.2 |
| Workloads are stated to include real coding and tool-using agents, not only | |
| single-turn generation. |
What the abstract does not give: any throughput figure, latency, tokens/sec, quantisation level, accuracy check against a datacenter baseline, or the specific hardware behind "gaming desktop" and "workstation GPU". Every number above is a capacity claim; none is a performance claim.
Significance
It is the strongest capacity datapoint this wiki holds for local serving of frontier open weights, and it names one: GLM-5.2 at 753B on a single workstation GPU — a model with its own page here (GLM-5.2) whose hosted pricing this repo checks daily against a provider range. That a model served commercially by 33 providers can be run on one machine is the concrete form of the argument Open-Weights Policy Fight tracks, and it is stated as a capability rather than an aspiration.
But the economics moved the other way in the same week, and the two belong on the same page. Consumer DDR5 prices are up as much as 485% year over year, with a 128GB kit at $3,399 — roughly 10× its lowest tracked price — and no forecast read expects relief before late 2027 (source). FreeToken's method is precisely to substitute host memory and bandwidth for GPU capacity. So the technique that makes a 753B model runnable at home lands in the quarter when the resource it spends became the scarce one. Neither source mentions the other; the pairing is this wiki's and is labelled as such.
The claim class matters. Fitting a model is not serving it usefully, and the abstract publishes no tokens/sec. On agentic workloads — the paper's own stated target — a rate below roughly conversational speed makes a long-horizon loop impractical regardless of capacity, and StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089) is the reminder that agent results are quoted in dollars and trials, neither of which this paper reports.
Open Questions
- What is the throughput? Not one rate figure appears in anything read, for any of the three hardware tiers.
- At what precision? A 753B model on one GPU implies aggressive quantisation, offload, or both, and the abstract names neither. The r/LocalLLaMA prefetch candidate on the same batch — "Petition to add a rule for people to add their DAMN quant levels to their posts" — is the community making the identical complaint.
- Is output quality preserved? No accuracy comparison against a datacenter deployment of the same weights was published.
- How much host RAM does each tier need? This is the figure that decides whether the DDR5 price move guts the result, and it is absent.
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
(2026). arXiv:2608.16157.