Attention Dilution
LARGE LANGUAGE MODELS
Attention dilution (also called context dilution) is one of the fundamental limitations of transformer-based LLMs when dealing with long contexts or extended agent memory.
Read articleBin Zhang's Field Notes
Modern notes for data, AI, and engineering systems.
Practical essays, implementation notes, and mental models for LLMs, infrastructure, data platforms, and the craft of building durable software.
Latest dispatch
Featured article
Fresh from the notebook
Recent writing
AI Terminology: Agents, Skills, RAG, MCP, and the Layers Beneath the Hype
AI ENGINEERING
How many of these terms do you actually recognize?
Read noteFrom Prompt to Response: A Step-by-Step Walkthrough of LLM Inference
LARGE LANGUAGE MODELS
From input to output, a prompt generally goes through seven steps: request packaging, tokenization, inference scheduling, prefill, and...
Read noteChatGPT in 2025: A Year in Review
LARGE LANGUAGE MODELS
ChatGPT Stats ChatGPT Growth ChatGPT Revenue
Read noteThe Mandate for Leadership in AI Engineering
AI ENGINEERING
Over the next 12 to 24 months, the differentiator among engineers will shift from mastery of programming languages like Rust, Go, or Python,...
Read noteLLM Interview Questions
LARGE LANGUAGE MODELS
Hyperparameters are external settings chosen before training, such as the learning rate or regularization strength.
Read noteLLM Training Epoch
LARGE LANGUAGE MODELS
As large language models (LLMs) scale up, researchers have begun to notice a growing imbalance between model size and the availability of...
Read note