KV-Centric LLM Serving: vLLM, SGLang, and Disaggregated Attention

The more I look at LLM serving, the more it feels like the main object is not the request, the model, or even the GPU.

The main object is the KV cache.

The old mental model is simple:

The newer serving stack looks more like this:

That is the thread connecting vLLM, SGLang, LMCache, Mooncake, Dynamo, DistServe, and many newer systems. They are not only optimizing attention math. They are managing attention state: where it lives, how it is reused, and how quickly it moves through the serving stack.

The Core Problem: Prefill and Decode Are Different

An autoregressive LLM request has two major phases:

These phases have very different shapes.

Prefill is compute-heavy because the whole input prompt can be processed in parallel. Long prompts, RAG documents, tool schemas, and few-shot examples all make this phase expensive.

Decode is sequential. Each new token depends on the previous token. The GPU repeatedly reads the existing KV cache, computes one more step, appends the new K/V, and repeats. This often underuses raw GPU compute while stressing memory bandwidth and latency.

When both phases run on the same GPU worker, they interfere with each other:

• Long prefills can block decode steps and create spikes in TTFT, or time to first token.
• Decode's low compute utilization can waste expensive GPU cycles.
• Prefill and decode compete for the same HBM memory that stores model weights and KV cache.
• One scheduling policy has to satisfy two very different workloads.

Prefill is parallel and compute-heavy; decode is sequential and memory-bandwidth-heavy. Running both on the same worker creates scheduling and memory contention.

This is the reason people talk about prefill/decode disaggregation.

The Solution Shape: Disaggregate the Serving Stack

Disaggregated serving splits prefill and decode onto separate worker pools:

Disaggregation lets prefill and decode scale separately, but it makes KV cache transfer and cache-aware routing part of the critical path.

The prefill cluster can be optimized for prompt throughput. The decode cluster can be optimized for token latency and concurrency. They can scale independently, use different parallelism strategies, and sometimes even use different GPU types.

That sounds clean, but it creates the hard new problem:

The KV cache produced by prefill must arrive at decode quickly enough that transfer does not erase the benefit of disaggregation.

So disaggregated serving is not just “split the workload.” It means the serving stack has to do several things at once:

• Split the workload between prefill-heavy and decode-heavy workers.
• Move KV cache efficiently so transfer latency does not erase the win.
• Route requests based on cache locality, not only on open worker slots.
• Manage cache tiers under memory pressure across GPU HBM, CPU DRAM, disk, and remote storage.

That is why modern systems increasingly feel KV-centric.

What the KV Cache Actually Stores

During generation, every token leaves behind key and value tensors.

The important detail is that KV is stored per layer. A token does not have one global K and one global V; it has a K/V pair at every transformer layer, and each pair is split by KV heads and head dimension.

Conceptually:

The rough memory formula is:

The 2 is for K and V.

For a Llama-2-7B-style model with multi-head attention:

That means KV cache grows with both context length and concurrency.

This table uses a standard MHA 7B-class example. Modern architectures reduce the slope:

The exact number matters. A 10K token prompt on a 70B-class model can mean many GB of KV transfer. With old-style MHA, it can be tens of GB. With a modern GQA model such as Llama-3-70B, it is much smaller, but still large enough to dominate latency if transferred poorly.

So the right lesson is not one fixed number. The lesson is:

KV cache size is determined by layers, KV heads, head dimension, dtype, context length, and concurrency.

The KV Transfer Problem

In a disaggregated stack, prefill creates KV on one worker and decode needs it on another.

That makes the transfer path a first-class part of the serving system:

Mooncake is a good example of this mental model. Its architecture treats KV cache as the central resource: a scheduler decides where work should go, and a transfer engine moves KV cache efficiently, using RDMA-style paths to reduce CPU involvement and overlap movement with compute where possible.

NVIDIA Dynamo makes the same issue visible in a practical serving stack. Its documentation separates the frontend, router, prefill workers, decode workers, and KV transfer path. It also warns that without RDMA or an equivalent fast fabric, KV movement can dominate TTFT and throughput.

The key point:

Disaggregation moves the bottleneck from “can this GPU do both phases?” to “can the system place and move KV cache fast enough?”

The Four-Tier KV Cache Hierarchy

Once KV cache becomes the main serving object, the system naturally becomes a memory hierarchy.

This hierarchy explains why different projects emphasize different things.

At L1, vLLM, SGLang, TensorRT-LLM, LMDeploy, and vAttention focus on making GPU-resident KV usable under high concurrency.

At L2 and below, LMCache, FlexGen, Dynamo KVBM, AttentionStore-style systems, and other KV stores start to matter. They ask: what happens when KV should outlive one GPU worker, one request, or one hot HBM allocation?

The serving engine wants hot KV in HBM, but HBM is scarce. A single GPU may have 40-80 GB; large model weights consume much of that before user KV cache even arrives. So every serious serving stack needs policies for:

• eviction
• promotion and demotion
• prefix reuse
• tenant isolation
• transfer scheduling
• cache-aware routing

This is why cache policy is not a minor implementation detail. It decides whether the stack keeps useful attention state close to compute or spends the whole time recomputing and moving it.

L1 GPU HBM: Where vLLM and SGLang Fit

vLLM and SGLang are easiest to understand as L1 KV-cache systems with different emphasis.

vLLM starts from memory allocation:

How do we store dynamic KV cache in GPU memory without wasting huge amounts of HBM?

SGLang starts from prefix reuse:

How do we automatically find shared prompt prefixes and reuse their KV cache across requests?

Both are attention-serving optimizations, but they sit at different levels.

vLLM: PagedAttention

vLLM’s core idea is PagedAttention.

The analogy is operating-system virtual memory. A process sees a logical address space, but the physical memory pages do not need to be contiguous. vLLM applies this idea to KV cache.

Instead of reserving one large contiguous KV tensor per request, vLLM splits KV cache into fixed-size blocks. A request has logical blocks, and the engine maps them to physical GPU-memory blocks.

The request still behaves as if its tokens are contiguous. The attention kernel knows how to gather the physical blocks.

This matters because request lengths are dynamic:

• One request may stop after 20 tokens.
• Another may continue for 4,000 tokens.
• A new request may arrive while older requests are decoding.
• Parallel sampling may share a prefix and then branch.

Without paging, the engine either over-reserves memory or suffers fragmentation. PagedAttention makes KV allocation granular and reusable.

On top of this block system, vLLM supports automatic prefix caching. It hashes full KV blocks using the parent block hash, tokens in the current block, and extra values such as LoRA IDs, multimodal hashes, or cache salts. If another request shares the same prefix blocks, vLLM can reuse the existing KV cache.

The practical implication is simple:

• identical stable prefixes help
• random timestamps near the front hurt
• inconsistent templates hurt
• block alignment matters because caching happens at block granularity

PagedAttention gives vLLM the memory substrate. Automatic prefix caching uses that substrate to avoid repeated prefill.

SGLang: RadixAttention

SGLang’s key idea is RadixAttention.

It stores reusable prompt and generation prefixes in a radix tree. A radix tree is like a compressed trie: edges can represent sequences of tokens rather than single tokens.

Conceptually:

root
+-- "system prompt + tool schemas"
|   +-- "user question A" -> cached KV
|   +-- "user question B" -> cached KV
+-- "few-shot examples"
    +-- "task A" -> cached KV
    +-- "task B" -> cached KV

When a new request arrives, SGLang looks for the longest matching prefix. If a matching prefix exists, the runtime reuses the corresponding KV cache and only computes the suffix.

This is especially useful for:

• shared system prompts
• repeated few-shot examples
• multi-turn conversations
• agent loops with stable tool context
• self-consistency sampling from the same prompt

RadixAttention is not a new attention formula. It is a runtime strategy for reusing attention state.

PagedAttention vs. RadixAttention vs. APC

These terms are easy to mix up, so I like this table:

The short version:

vLLM made KV memory behave more like paged virtual memory. SGLang made repeated prompt programs behave more like prefix-tree lookups.

They are not competing mental models. They are different layers of the same KV-cache problem.

Framework Coverage by Cache Tier

This table is not meant to be a strict feature matrix. It is a mental map of where each system tends to focus.

The important correction to my own thinking: “framework support” is not binary. A serving engine may own L1, while a cache layer or distributed runtime owns L2-L4. In production these are composed.

Metrics Improved by Disaggregation

Disaggregation is valuable only if it improves the metrics users and operators care about.

DistServe calls the real target goodput: how much traffic can be served while meeting both TTFT and TPOT constraints. That is a better target than raw throughput because a system that produces many tokens but violates latency SLOs is not actually good serving.

Major Systems and Their Approach

Here is the landscape from my notes, rewritten as a clearer map.

Mooncake

Mooncake is a KV-cache-centric disaggregated serving architecture used for Moonshot AI’s Kimi service.

The useful mental split:

• Conductor: global scheduler
• Transfer Engine: RDMA-based KV movement
• KV-centric cache: treats KV cache as the first-class cluster resource

Mooncake is especially interesting for long-context workloads because repeated large prompts make KV placement and reuse extremely valuable.

NVIDIA Dynamo

Dynamo is a distributed inference serving stack with:

• frontend and router components
• backend workers for vLLM, SGLang, and TensorRT-LLM
• KV-cache-aware routing
• disaggregated serving
• KV cache offloading integrations

The key term in Dynamo is NIXL, a transfer abstraction around paths such as NVLink, InfiniBand, and PCIe. The router can make decisions based on KV locality rather than only worker load.

DistServe

DistServe formalized the prefill/decode split as a goodput optimization problem.

It argues that prefill and decode should be placed and parallelized separately because they have different latency targets:

• prefill affects TTFT
• decode affects TPOT

The core lesson is that disaggregation is not only about utilization. It is about meeting both latency constraints at the same time.

vLLM Disaggregated Prefill

vLLM has experimental disaggregated prefilling support and examples that separate prefill and decode workers. The key abstraction is a transfer connector: the runtime needs a way to move KV state from the prefill side to the decode side.

In practice, this area connects vLLM to systems such as LMCache, NIXL/Dynamo-style transfer, and Mooncake-style serving architectures.

Splitwise

Splitwise explores heterogeneous serving: use different hardware for different phases.

The motivation is intuitive:

• prefill may benefit from high-end compute-heavy GPUs
• decode may run cost-effectively on different GPUs if memory and latency targets are satisfied

This matters because disaggregation is not only a latency trick. It can also be a cost-structure trick.

Framework Landscape by Problem

Another way to organize the ecosystem is by the problem each system is trying to solve.

This table is the bigger picture behind the vLLM/SGLang comparison. vLLM and SGLang are not isolated tools; they are two important nodes in a larger KV-management ecosystem.

Application Design Still Matters

Infrastructure can only reuse what the application gives it.

The engine sees tokens, not your intent. If two prompts do not tokenize to the same prefix, the cache cannot help.

Final Mental Model

The attention formula tells us what the model computes:

The serving stack asks a different set of questions:

• Where is the KV cache?
• Is the useful prefix already cached?
• Is it in GPU HBM, CPU DRAM, NVMe, or remote storage?
• Should this request go to the least busy worker or the worker with the best KV locality?
• Is it cheaper to recompute KV or transfer it?
• Which KV blocks should be evicted?

That is the shift:

Modern LLM serving is becoming KV-cache management.

vLLM’s PagedAttention solves the hot HBM allocation problem. vLLM’s automatic prefix caching reuses repeated block prefixes. SGLang’s RadixAttention organizes repeated prompt paths for runtime reuse. Disaggregated systems such as Mooncake, Dynamo, and DistServe move the same idea up to the cluster level.

A rough cost instinct is:

It is not a pricing equation. It is a compass.

The goal is to compute less repeated KV, move fewer KV bytes, keep hot KV close to the GPU, and schedule requests around cache locality. Once that clicks, vLLM attention, SGLang attention, LMCache, and disaggregated serving stop feeling like separate topics.

They are all different layers of the same system:

A system for managing expensive attention state.