For years, large language model (LLM) inference has effectively been confined to a single location. The high-bandwidth RDMA networks that modern serving relies on have kept both the prefill and decode stages inside one datacenter, and often within a single rack. A paper from Moonshot AI and researchers at Tsinghua University argues that this constraint is beginning to lift, and proposes an architecture designed to take advantage of the change. The work, titled Prefill-as-a-Service, was posted to arXiv in April 2026.
Prefill-as-a-Service (PrfaaS) is a cross-datacenter serving architecture that selectively offloads long-context prefill work to a standalone, compute-intensive prefill cluster, then transfers the resulting KVCache over commodity Ethernet to a local prefill-decode (PD) cluster for the decode stage. In a case study on a large internal hybrid-attention model, the design is reported to deliver 54% higher serving throughput than a homogeneous PD baseline and 32% higher than a naive heterogeneous setup, while using only a fraction of the available cross-datacenter bandwidth.
Why conventional serving stays inside one datacenter
The bottleneck is the size and speed of the KVCache. With conventional Grouped Query Attention (GQA) models, a long-context request can generate KVCache on the order of tens of gigabits per second on a single multi-GPU instance. Transferring that volume without stalling the compute requires RDMA-class interconnects, which is why traditional prefill-decode separation has been tied to a single datacenter-scale network fabric. Moving prefill and decode onto separate clusters, let alone separate datacenters, has not been practical under that constraint.
How hybrid attention changes the math
What makes PrfaaS timely is an architectural shift at the model level. A growing class of models adopts hybrid attention stacks that combine many linear-complexity or bounded-state layers, such as linear attention, Multi-Head Latent Attention (MLA), and Sliding Window Attention (SWA), with only a small number of full-attention layers. Because just those few full-attention layers produce a large, length-dependent KVCache, the total cache that must be moved shrinks dramatically. The paper reports that hybrid attention can reduce KVCache size by up to roughly 36 times compared with a comparable GQA model, which is what brings cross-datacenter transfer over ordinary Ethernet within reach.
How requests are routed
PrfaaS separates two kinds of cache into a unified block pool: the compact, fixed-size cache from linear-style layers and the block-level, linearly growing cache from full-attention layers that supports partial prefix matching. Routing is governed by a length-based threshold. After subtracting any cached prefixes, requests whose remaining prefill length exceeds a set threshold are sent to the PrfaaS cluster, and their KVCache is forwarded over Ethernet to a decode node; shorter requests stay on the local PD path. In the case study, the throughput-optimal threshold of around 19,400 tokens routed roughly half of all requests, the longest ones, to the PrfaaS cluster.
Making the Ethernet path reliable takes more than low KVCache volume. The authors describe several transport mechanisms, including layer-wise prefill pipelining that overlaps KVCache generation with transmission, multi-connection TCP transport to make full use of available bandwidth, and scheduling that keeps the system near its throughput-optimal operating point.
What the numbers show
In the case study, a PrfaaS cluster of 32 H200 GPUs was paired with a local PD cluster of 64 H20 GPUs over a VPC network providing roughly 100 Gbps of cross-cluster bandwidth. Under the optimal configuration, total PrfaaS egress was about 13 Gbps, only around 13% of the available Ethernet capacity, and the prefill cluster remained compute-bound with bandwidth to spare. The authors project that the approach scales: even for a 10,000-GPU datacenter, the egress bandwidth required for KVCache transfer is estimated at about 1.8 Tbps, well within the capacity of modern inter-datacenter links. Mean time to first token (TTFT) fell by about 50% and P90 TTFT by about 64% relative to the homogeneous baseline.
Limitations and what to watch
The headline figures come from a single internal case study on one hybrid-attention model and one hardware pairing, so results on other models, request mixes, and network conditions may differ. The benefits depend heavily on the move toward hybrid attention; serving a conventional GQA model this way would still demand far more bandwidth. Performance is also sensitive to the routing threshold and to real-world network variability such as latency spikes and packet loss across datacenters, which a controlled benchmark may understate. The broader case is likely to strengthen as context windows grow, KVCache compression matures, and stage-specific hardware, such as NVIDIA’s Rubin CPX for prefill, becomes more widely available, but independent reproduction across diverse production settings will be the real test. Further coverage of the study is available from MarkTechPost, and related discussion of where AI infrastructure value is shifting appears in this analysis of control over the AI stack.
