LLM Inference & Serving

Note 01 1 of 4

Mar 7, 2026 LARGE LANGUAGE MODELS

From Prompt to Response: A Step-by-Step Walkthrough of LLM Inference

Update: For a deeper systems-level treatment of LLM inference, especially the interaction between request scheduling, prefill, decode, and KV-cache reuse, see arXiv:2606.24937.

llm fundamentals

Read

Note 02 2 of 4

Mar 15, 2026 LARGE LANGUAGE MODELS

Attention Dilution

Attention dilution (also called context dilution) is one of the fundamental limitations of transformer-based LLMs when dealing with long contexts or extended agent memory.

llm fundamentalsrag

Read

Note 03 3 of 4

Jul 5, 2026 AI ENGINEERINGLARGE LANGUAGE MODELS

KV-Centric LLM Serving: vLLM, SGLang, and Disaggregated Attention

The more I look at LLM serving, the more it feels like the main object is not the request, the model, or even the GPU.

vllmsglangpaged attentionradix attentionkv cachedisaggregated servingprefill decode

Read

Note 04 4 of 4

Jul 3, 2026 AI ENGINEERINGSOFTWARE ENGINEERING

Reproducing vLLM and LMCache KV Cache Reuse on a CPU-Only MacBook

I became interested in LMCache because it sits in the part of LLM serving that feels both very practical and very under-discussed: KV cache movement.

vllmlmcachekv cachecpuuv

Read