EN한
← wiki

$ cat wiki/papers/2026/2611.12304-evolving-infra.md

Evolving Infra: agent-adaptive serving

paperupdated 2026-11-16created 2026-11-16

TL;DR

Evolving Infra adapts compute allocation and KV cache management to changing agent workloads. The authors report lower request latency and time to first token than Dynamo in their tested configurations. (source)

Authors & Org

Hao Kang, Weili Xu, Ziyang Li, Zheng Du, Hanchen Li, Tian Xia, Song Bian, Tushar Krishna, Huizi Mao, Ion Stoica and Song Han. Affiliations are not established by the reviewed abstract record. (source)

Method

The system separates compute allocation from KV cache management so nodes can switch between prefill and decode. An online planner adjusts the prefill/decode split to workload composition; a distributed KV manager uses signals from agent context management to guide cache eviction and transfer. (source)

Results

Across coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, the authors report reductions in mean end-to-end request latency and P95 TTFT of up to 4.1× and 14×, respectively, versus NVIDIA Dynamo. These are different latency measures and reported maxima, not a uniform speedup or measured dollar saving. (source)

Significance

Interpretation: agent context behavior can inform serving decisions, connecting Agents (LLM Agents) with runtime resource allocation. The contribution concerns infrastructure adaptation rather than a demonstrated improvement in model reasoning. (source)

Open Questions

The reviewed abstract does not establish which workload reaches each maximum, a total-cost comparison, or performance on smaller deployments. Full experiments and implementation were not reviewed or reproduced. (source)

Cite

Referenced by

Sources