Plain-language summary: modern GPUs are often too powerful for the small AI models running on them, leaving much of the chip idle. Engineers at Databricks tested an NVIDIA feature called MPS that lets several model instances share one GPU, and found it can boost throughput substantially — but only for very small models with short inputs. This article summarizes their findings.
Smaller language models are becoming increasingly capable and applicable to a wide range of enterprise use cases, while each new GPU generation packs dramatically more compute and memory bandwidth. The result: even under highly concurrent workloads, small LLMs often leave a large portion of GPU compute and memory bandwidth idle.
Databricks serves many such small models in production — for code completion, retrieval, grammar correction and other specialized tasks — and its engineering team set out to rigorously test whether NVIDIA’s Multi-Process Service (MPS) delivers higher throughput per GPU in a production environment. The team’s published study found that MPS provides meaningful throughput wins in specific settings: very small language models (roughly 3B parameters or fewer) with small-to-medium context lengths, and inference engines with significant CPU overhead.
The explanation is twofold. At the GPU level, MPS enables meaningful kernel overlap when individual engines leave compute or memory bandwidth under-utilized — especially during attention-heavy phases in smaller models. As a useful side effect, it can also reduce CPU bottlenecks, such as scheduler or image-preprocessing overhead in multimodal workloads, by splitting the total batch across engines and reducing the CPU load per engine.
What is MPS?
NVIDIA’s Multi-Process Service is a binary-compatible implementation of the CUDA API that allows multiple processes to share a GPU more efficiently. Instead of processes taking turns (leaving the GPU idle between turns), their kernel and memory operations are multiplexed and overlapped by the MPS server — effectively squeezing more work out of the same hardware.
The scaling study
The experiments compared two identical inference engines running on the same NVIDIA H100 GPU with MPS enabled against a single-engine baseline, using a fully balanced, homogeneous workload.
Key observations from the scaling study: MPS provided more than 50% throughput uplift for small models with short contexts; the benefit falls roughly log-linearly as context length increases for a given model size; and as model size grows, the benefit shrinks rapidly even at short contexts. Around the 7B-model / 2K-context point, gains fall below 10% and eventually turn into a regression.
For prefill-heavy workloads the pattern was similar: small models benefited, medium-sized models (~3B) saw diminishing returns as context grew — eventually degrading performance — and models larger than 3B saw no benefit at all. Overall, MPS pays off most in low-utilization setups: small models and compact contexts, where kernels leave room to overlap.
Where do the gains actually come from?
To pinpoint the cause, the team broke the transformer down into its two main building blocks — MLP layers and attention — and isolated each, removing complicating factors such as CPU overhead.
GPU resources required | |||
| n = reference length | prefill(calculation) | Decode (Memory Bandwidth) | decode (calculate) |
| mlp | But) | O(1) | O(1) |
| pay attention | O(n^2) | But) | But) |
The two components scale differently. An MLP layer loads its weights once and processes each token independently, consuming a constant amount of memory bandwidth and compute per token. Attention, by contrast, loads the KV cache and computes dot products against all previous tokens, so its bandwidth and compute needs grow with context length.
MLP-only ablation (attention removed)
The MLP-only gains were modest and quickly disappeared. As model size or context length increases, a single engine already saturates the computation — more FLOPs per token in larger MLPs, more tokens with longer sequences. Once an engine is compute-bound, running two saturated engines side by side adds almost nothing.
Attention-only ablation (MLP removed)
Using Qwen2.5-3B, the team then measured the attention-only configuration analogously.
The results were notable: the attention-only workload showed a substantially larger MPS gain than the full model, for both prefill and decode. For decode, the benefit declined linearly with context length — consistent with attention’s resource needs growing as the context grows. For prefill, the gain dropped off faster.
Comparing the full model’s actual gain against a weighted average of the isolated components suggests most of the benefit comes from overlapping two under-saturated attention phases, with little added from attention-MLP overlap: when one engine is running saturated MLP kernels, the other engine’s unsaturated attention kernels have less opportunity to slot in.
Bonus benefit: recovering GPU time lost to CPU overhead
The most severe form of under-utilization occurs when the GPU sits idle waiting on CPU work — scheduling, tokenization, or image preprocessing in multimodal models. In a single-engine setup, those CPU stalls directly waste GPU cycles. With MPS, whenever one engine is blocked on the CPU, the other can take over the GPU, turning dead time into productive computation.
To isolate this effect, the team deliberately chose a regime where the GPU-level advantages disappear: Gemma-4B at a size and context length where attention and MLP are already well saturated.
In that setting, the single-engine baseline was limited by scheduler overhead. The study reports that enabling asynchronous scheduling in vLLM lifted throughput by about 33% — and that MPS captured a similar benefit in environments where such engine-level optimizations are not available.
Limitations and what to watch
MPS is a scalpel, not a silver bullet. Outside the sweet spots — for example 7B+ models, contexts beyond a few thousand tokens, or already compute-bound workloads — GPU-level gains largely vanish. The study also catalogs real operational costs: extra moving parts (the MPS daemon, client environment setup, and a router to balance traffic across engines); harder debugging, since engines are not isolated and a memory leak or out-of-memory error in one can take down the others; a heavier monitoring burden; and a critical failure mode in which one misbehaving client can corrupt or starve the entire shared GPU. The results are also specific to the tested hardware (H100) and engine stack — teams should benchmark their own workloads, and watch engine-native features such as asynchronous scheduling, which can capture some of the same gains with less complexity. For organizations weighing this kind of infrastructure tuning, the trade-offs echo broader questions about operating AI systems under tight resource budgets.
The bottom line
For fleets of very small models with short contexts — or engines dragged down by CPU overhead — MPS can deliver double-digit to 50%+ throughput gains on the same hardware. For everything else, it is added complexity with little return. The Databricks study, authored by Xiaotong Jiang, stands as a useful reminder that in the small-model era, much of the untapped cost-efficiency in AI serving lives below the model layer.
