\n Try it: click quick — the query now matches quick's Key, so most attention\n flows there (and a little to fox, whose Key shares that direction). The output\n vector becomes mostly quick's Value. That is all attention does: score by Key match, blend the\n Values. Real attention divides scores by √dₖ before softmax; omitted here for legibility.\n
\n \n );\n}\n",
+ "frameTitle": "Attention Explorer",
+ "frameCaption": "A toy 3-dimensional attention with real math running in your browser. Drag the query, or click a token to point the query at its Key — the dot-product scores, softmax weights, and blended output all update live.",
+ "frameSize": "wide",
+ "frameExpand": true
+ },
+ "children": []
+ },
+ {
+ "id": "note-35",
+ "type": "note",
+ "props": {},
+ "content": [
+ {
+ "type": "text",
+ "text": "Real models run attention several times in parallel — ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "multi-head attention",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". Each ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "head",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " gets its own slice of the vector and its own Q/K/V, so different heads can specialize: one tracks grammar, another long-range topic, another position. Llama 3 70B has ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "64 query heads",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " of dimension 128 (64 × 128 = 8192). But it deliberately shares Keys and Values across groups of heads — only ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "8 KV heads",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — so there’s less K/V to store. That’s exactly the ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "8",
+ "styles": {
+ "code": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " in the KV-cache formula later, and the trick has a name: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "grouped-query attention (GQA)",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". It’s your first glimpse of an optimization baked right into the model to shrink the cache.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-36",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The root cause: generating text one token at a time",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-37",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Here’s where memory enters, and it’s the whole point. An LLM does ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "not",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " write a sentence at once. It’s ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "autoregressive",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ": it generates ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "one token, appends it to the input, and runs the entire model again",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " to get the next one. Like predictive text that feeds its own output back in, forever.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "codeBlock-38",
+ "type": "codeBlock",
+ "props": {
+ "language": "text"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": " \"The quick brown\" → model → \"fox\"\n \"The quick brown fox\" → model → \"jumps\"\n \"The quick brown fox jumps\" → model → \"over\"\n ...",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-39",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Now connect that to attention. To generate the next token, its Query must be scored against the Keys and Values of ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "every token so far",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". Token 1000 attends over 999 predecessors. Which raises the obvious, expensive question: at each step, do we recompute K and V for the entire history?",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-40",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "We could — and it would be catastrophic. Recomputing all past K/V every step is O(n²) work over a sequence. So instead we do the one optimization that makes generation viable at all: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "we compute each token’s K and V once, and cache them.",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " New token arrives → compute its K and V → ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "append to the cache",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " → done. The Keys and Values of old tokens never change, so there’s no reason to ever recompute them.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "svg-41",
+ "type": "svg",
+ "props": {
+ "code": "",
+ "caption": "Autoregressive decoding: each new token computes its Key and Value once and appends them to the cache. The cache only ever grows — every generated token adds one more K/V pair, at every layer.",
+ "width": 620
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-42",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "And ",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": "there",
+ "styles": {
+ "bold": true,
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " is your root cause.",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " The cache grows by one K/V pair ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "per token, per layer, per head",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — and it never shrinks during a request, because every future token might still want to attend to it. That’s the whole answer to “why does memory keep getting bigger over each iteration”:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-43",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Weights are constant",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — 140 GB whether you generate 1 token or 100,000. Loaded once, shared by everyone.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-44",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The KV cache is the opposite",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — it starts near zero and grows linearly with every token generated, and it’s ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "private to each request",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ".",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-45",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Put real numbers on it for Llama 3 70B: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "2 (K,V) × 80 layers × 8 KV-heads × 128 head-dim × 2 bytes ≈ 320 KB per token",
+ "styles": {
+ "code": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". At 32K tokens → ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "10 GB",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", for one conversation. Now you know ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "why",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ": it’s 320 KB accumulated 32,768 times, one autoregressive step at a time.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-46",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Here’s that formula as a live calculator — pick a model, drag the context length and the number of concurrent users, and watch the KV cache fill the GPU against the fixed weights (it’s the Red Hat slide that started this, except now you can break it yourself):",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "customComponent-47",
+ "type": "customComponent",
+ "props": {
+ "componentName": "KVCacheGrower",
+ "source": "import { useState, useMemo, type CSSProperties, type ReactNode } from 'react';\n\n// Real KV-cache math, computed live in the browser.\n// KV bytes = 2 (K,V) × layers × kv_heads × head_dim × precision_bytes × tokens × requests\n// Weights bytes = params × precision_bytes\n\ntype Model = {\n name: string;\n layers: number;\n kvHeads: number;\n headDim: number;\n params: number; // total parameters\n};\n\nconst MODELS: Model[] = [\n { name: 'Llama 3 8B', layers: 32, kvHeads: 8, headDim: 128, params: 8.0e9 },\n { name: 'Llama 3 70B', layers: 80, kvHeads: 8, headDim: 128, params: 70.6e9 },\n { name: 'Llama 3 405B', layers: 126, kvHeads: 8, headDim: 128, params: 405e9 },\n];\n\nconst NODES = [80, 160, 320, 640]; // GB of GPU memory (multiples of an 80GB card)\nconst GiB = 1024 ** 3;\n\nconst mono: CSSProperties = { fontFamily: 'var(--font-mono)', fontSize: 12, color: 'var(--ink-2)' };\n\nfunction Stat({ label, value, accent }: { label: string; value: string; accent?: boolean }) {\n return (\n \n
\n ⚠ Over capacity — weights + KV exceed {nodeGB} GB. Reduce users/context, use FP8, or add GPUs.\n
\n )}\n\n {/* Stats */}\n
\n \n \n \n \n
\n\n \n per token = 2 × {m.layers} layers × {m.kvHeads} kv-heads × {m.headDim} × {precBytes} B ={' '}\n {calc.perTokenKB.toFixed(0)} KB · × {ctx.toLocaleString()} tok ={' '}\n {calc.perRequestGB.toFixed(1)} GB per request\n \n\n
\n Try it: pick Llama 3 70B, 32K context — one request is ~10 GB. Now drag users up:\n the KV cache races past the weights and fills the node. Switch to FP8 and it halves.\n That single bar is why prefix caching, paged-KV, GQA, and quantized-KV all exist.\n
\n \n );\n}\n",
+ "frameTitle": "KV-Cache Grower",
+ "frameCaption": "Every number is the real formula, computed live. Try Llama 3 70B at 32K context — one request is ~10 GB — then drag the user count up and watch the cache overrun the node.",
+ "frameSize": "normal",
+ "frameExpand": true
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-48",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Drag it around and the scaling speaks for itself. Here is the same model at the context lengths people actually deploy:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "table-49",
+ "type": "table",
+ "props": {
+ "textColor": "default"
+ },
+ "content": {
+ "type": "tableContent",
+ "columnWidths": [
+ null,
+ null,
+ null
+ ],
+ "rows": [
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Context",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "What it is",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "KV cache (one request)",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "2K",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "one chat turn",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "640 MB",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "8K",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "standard",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "2.5 GB",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "32K",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "long document / codebase",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "10 GB",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ },
+ {
+ "cells": [
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "128K",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "Llama 3’s maximum",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ },
+ {
+ "type": "tableCell",
+ "content": [
+ {
+ "type": "text",
+ "text": "40 GB",
+ "styles": {}
+ }
+ ],
+ "props": {
+ "colspan": 1,
+ "rowspan": 1,
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ }
+ }
+ ]
+ }
+ ]
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-50",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Sit with that last row. A single 128K-context request needs ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "40 GB of KV cache — nearly a third of the model’s entire 140 GB of weights — for one user’s conversation.",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " Now serve ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "10",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " such users at once: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "400+ GB of KV cache alone",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", on top of the model. That is more than the ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "entire",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " memory of a 4×80 GB node. This — not the math, not the weights — is why managing the KV cache is the single biggest job of a production inference server.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-51",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Where does all this live? The GPU memory hierarchy",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-52",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "We’ve been saying “GPU memory” as if it’s one thing. It isn’t — and its structure is the last piece that makes ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "why data movement dominates",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " finally click. A GPU has three tiers, and they obey the same law as every memory system you already know, from a database’s buffer pool to your CPU’s caches: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "the closer to the compute, the faster — and the smaller.",
+ "styles": {
+ "bold": true
+ }
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "svg-53",
+ "type": "svg",
+ "props": {
+ "code": "",
+ "caption": "The GPU memory hierarchy — same shape as CPU caches. Weights and the KV cache sit in HBM; the tensor cores compute out of tiny, blazing-fast SRAM. Figures are for an NVIDIA A100.",
+ "width": 620
+ },
+ "children": []
+ },
+ {
+ "id": "paragraph-54",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "If you know CPU caches, you already know this shape: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "SRAM is the L1 cache",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — tiny, on-chip, sitting right next to the compute units (the ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "tensor cores",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ") and feeding them at a staggering ~19 TB/s. ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "HBM is main RAM",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — gigabytes, but an order of magnitude slower. ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "Host DRAM is the far-off disk",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", reached over a thin ~12 GB/s cable. Now place our two tenants:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-55",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Model weights (140 GB)",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " load into ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "HBM",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " once at startup and stay there for the server’s life.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-56",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The KV cache",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " also lives in ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "HBM",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", growing per request — which is why ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "HBM capacity",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ", not compute, caps how many users you can serve.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-57",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Every forward pass, small chunks of weights and KV are pulled ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "HBM → SRAM",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " for the tensor cores to work on, then results are written back.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-58",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "And there’s the physical reason decode is ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "memory-bound",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ": the tensor cores are so fast they finish and sit idle, ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "waiting for the next chunk to arrive over the HBM→SRAM path",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". The bottleneck isn’t the arithmetic — it’s the data movement.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-59",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "This also finally explains ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "why we cache K and V but not Q.",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " Tensors come in two kinds. ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "Persistent",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " ones — the weights and the KV cache — stay in HBM because they’re needed again and again. ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "Transient",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " ones — a token’s Q, its freshly-computed K and V, the attention output, the FFN output — are born in SRAM, used immediately, and discarded. Q is transient (used once, the moment its token does the looking); K and V get ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "promoted",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " into the persistent cache precisely because every future token will still need them.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "note-60",
+ "type": "note",
+ "props": {},
+ "content": [
+ {
+ "type": "text",
+ "text": "Vocabulary we’ll lean on from here: the umbrella term for any array of numbers — a single value, a vector, a matrix, or higher — is a ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "tensor",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ". Weights, Q/K/V, and the KV cache are all tensors. “Data moving through the model” just means tensors moving between these memory tiers.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-61",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "That leaves one sentence the entire optimization field hangs on: ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "every technique you’ll meet is one of three moves — move less data, move it more efficiently, or manage the memory better.",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " Quantization shrinks the bytes; paged KV manages the cache; FlashAttention keeps more of the work inside SRAM. Same three levers, over and over.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-62",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Why this is the hinge of the entire field",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-63",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Step back and the two costs you now understand explain nearly every inference technique in existence:",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-64",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Memory-bound decode",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — each step drags all 140 GB of weights through the chip to compute ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "one",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " token. Barely any math per byte. That’s why decode is slow and why batching many users’ tokens together is the throughput lever. (A dedicated post on this — the roofline model — is coming.)",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "bulletListItem-65",
+ "type": "bulletListItem",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "The growing KV cache",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — caps how many users fit in memory at once, and makes long context expensive. That’s why ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "paged KV cache, quantized KV, GQA, MLA, prefix sharing",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " all exist. Every one is a strategy to store, shrink, or reuse this cache.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-66",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "This is why memory dominates every conversation about inference, and the reason is structural: an autoregressive transformer is a machine that re-reads all of its weights and its entire growing memory to produce one token at a time. The whole field of inference optimization is the collected set of tricks for making that bearable.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "heading-67",
+ "type": "heading",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left",
+ "level": 1,
+ "isToggleable": false
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "What comes next",
+ "styles": {}
+ }
+ ],
+ "children": []
+ },
+ {
+ "id": "paragraph-68",
+ "type": "paragraph",
+ "props": {
+ "backgroundColor": "default",
+ "textColor": "default",
+ "textAlignment": "left"
+ },
+ "content": [
+ {
+ "type": "text",
+ "text": "Everything downstream builds on the two costs in this post — and the highest-leverage trick attacks the bigger one, the KV cache. Under naive allocation, most of a request’s reserved cache sits ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "unused",
+ "styles": {
+ "italic": true
+ }
+ },
+ {
+ "type": "text",
+ "text": ": you set aside room for the full context length, but a typical request never fills it. ",
+ "styles": {}
+ },
+ {
+ "type": "text",
+ "text": "Paged KV cache",
+ "styles": {
+ "bold": true
+ }
+ },
+ {
+ "type": "text",
+ "text": " — the idea behind vLLM’s PagedAttention — reclaims that waste with a technique borrowed straight from operating-system memory paging. That’s where this series goes next.",
+ "styles": {}
+ }
+ ],
+ "children": []
+ }
+ ],
+ "tableVariants": {
+ "table-49": {
+ "border": "lined",
+ "zebra": true,
+ "caption": "Llama 3 70B KV cache per request (FP16). It grows linearly with context — and every row is on top of the fixed 140 GB of weights."
+ }
+ }
+}
diff --git a/src/content/posts/how-transformers-work-with-attention-and-kv-cache/AttentionExplorer.tsx b/src/content/posts/how-transformers-work-with-attention-and-kv-cache/AttentionExplorer.tsx
new file mode 100644
index 0000000..c402f92
--- /dev/null
+++ b/src/content/posts/how-transformers-work-with-attention-and-kv-cache/AttentionExplorer.tsx
@@ -0,0 +1,206 @@
+import { useState, useMemo, type CSSProperties, type ReactNode } from 'react';
+
+// A toy 3-dimensional attention. The MECHANISM is exactly the real one —
+// dot-product scores, softmax, value blend — just small enough to see every number.
+// Four context tokens, each with a fixed Key (its "label") and Value (its "content").
+// You control the Query. Watch attention flow to whichever Key the Query points at.
+
+type Vec = [number, number, number];
+
+const TOKENS = ['The', 'quick', 'brown', 'fox'] as const;
+
+// Key = "what I contain / my label" (near-orthogonal so a query can select one)
+const KEYS: Vec[] = [
+ [1.0, 0.1, 0.1], // The
+ [0.1, 1.0, 0.1], // quick
+ [0.1, 0.1, 1.0], // brown
+ [0.7, 0.7, 0.2], // fox — shares direction with "quick", so it lights up too
+];
+
+// Value = "my actual content to hand over"
+const VALUES: Vec[] = [
+ [1.0, 0.0, 0.0], // The
+ [0.0, 1.0, 0.0], // quick
+ [0.0, 0.0, 1.0], // brown
+ [0.5, 0.5, 0.5], // fox
+];
+
+const dot = (a: Vec, b: Vec) => a[0] * b[0] + a[1] * b[1] + a[2] * b[2];
+
+function softmax(xs: number[]): number[] {
+ const m = Math.max(...xs);
+ const exps = xs.map((x) => Math.exp(x - m));
+ const sum = exps.reduce((a, b) => a + b, 0);
+ return exps.map((e) => e / sum);
+}
+
+const mono: CSSProperties = { fontFamily: 'var(--font-mono)', fontSize: 12, color: 'var(--ink-2)' };
+const fmtVec = (v: Vec) => `[${v.map((n) => n.toFixed(2)).join(', ')}]`;
+
+function Slider({
+ label,
+ value,
+ onChange,
+}: {
+ label: string;
+ value: number;
+ onChange: (v: number) => void;
+}) {
+ return (
+
+ );
+}
+
+export default function AttentionExplorer(): ReactNode {
+ const [q, setQ] = useState([0.2, 0.9, 0.1]);
+
+ const { scores, weights, output } = useMemo(() => {
+ const scores = KEYS.map((k) => dot(q, k));
+ const weights = softmax(scores);
+ const output: Vec = [0, 0, 0];
+ VALUES.forEach((v, i) => {
+ output[0] += weights[i] * v[0];
+ output[1] += weights[i] * v[1];
+ output[2] += weights[i] * v[2];
+ });
+ return { scores, weights, output };
+ }, [q]);
+
+ const topIdx = weights.indexOf(Math.max(...weights));
+ const near = (a: number, b: number) => Math.abs(a - b) < 0.05;
+ const mimics = (i: number) =>
+ near(q[0], KEYS[i][0]) && near(q[1], KEYS[i][1]) && near(q[2], KEYS[i][2]);
+
+ return (
+
+
+ Attention is a search
+
+
+ {/* Query controls */}
+
+ Query vector (what this token is looking for): {fmtVec(q)}
+
+ Try it: click quick — the query now matches quick's Key, so most attention
+ flows there (and a little to fox, whose Key shares that direction). The output
+ vector becomes mostly quick's Value. That is all attention does: score by Key match, blend the
+ Values. Real attention divides scores by √dₖ before softmax; omitted here for legibility.
+
+ ⚠ Over capacity — weights + KV exceed {nodeGB} GB. Reduce users/context, use FP8, or add GPUs.
+
+ )}
+
+ {/* Stats */}
+
+
+
+
+
+
+
+
+ per token = 2 × {m.layers} layers × {m.kvHeads} kv-heads × {m.headDim} × {precBytes} B ={' '}
+ {calc.perTokenKB.toFixed(0)} KB · × {ctx.toLocaleString()} tok ={' '}
+ {calc.perRequestGB.toFixed(1)} GB per request
+
+
+
+ Try it: pick Llama 3 70B, 32K context — one request is ~10 GB. Now drag users up:
+ the KV cache races past the weights and fills the node. Switch to FP8 and it halves.
+ That single bar is why prefix caching, paged-KV, GQA, and quantized-KV all exist.
+
+
+ );
+}
diff --git a/src/content/posts/how-transformers-work-with-attention-and-kv-cache/index.mdx b/src/content/posts/how-transformers-work-with-attention-and-kv-cache/index.mdx
new file mode 100644
index 0000000..629dac1
--- /dev/null
+++ b/src/content/posts/how-transformers-work-with-attention-and-kv-cache/index.mdx
@@ -0,0 +1,353 @@
+---
+title: 'How a Transformer Really Works: Attention, the KV Cache, and Why Inference Eats Memory'
+summary: 'A from-scratch tour of what''s actually inside an LLM: how a transformer turns tokens into predictions, what Query, Key, and Value really mean, and how generating text one token at a time builds the KV cache — the growing pool of memory that makes inference so expensive.'
+authors: ['dinesh']
+date: '2026-07-21'
+updated: '2026-07-21'
+readMin: 10
+topic: 'Architecture'
+topicId: 'architecture'
+tags: ['transformer', 'attention', 'qkv', 'kv-cache', 'autoregressive']
+---
+
+import AttentionExplorer from './AttentionExplorer';
+import Interactive from '../../../components/Interactive';
+import KVCacheGrower from './KVCacheGrower';
+
+In [Neural Networks From Zero](/blog/neural-networks-from-zero) we built the machine from one neuron up and learned to size it: a 70B model is **140 GB of weights**. But running it needs _more_ memory than that — a second pool that starts near zero and grows with every token of the conversation, until one long request can cost **\~10 GB on its own**. Why does _one user’s_ conversation cost 10 GB? Why do people obsess over inference memory? Why does long context get so expensive?
+
+You can’t answer any of that from the outside. The cost is a _consequence_ of how a transformer actually works — specifically, how it generates text one token at a time. So this post opens the box. By the end you’ll understand the architecture, what Q/K/V genuinely mean, and — the payoff — exactly _why memory grows with every token you generate._ That growth is the root cause the whole optimization field is built around.
+
+## The map: what’s actually stacked inside
+
+Strip away the mystique and a transformer is a sandwich you already half-understand. Text comes in, gets turned into vectors (the [numbers we talked about](/blog/neural-networks-from-zero)), flows through the **same block repeated N times**, and comes out as a prediction for the next token.
+
+
+{/* mermaid
+flowchart TB
+ P["`**Prompt**
+'The quick brown'`"] --> E["`**Embeddings**
+one vector per token`"]
+ E --> BLK
+ subgraph BLK["Transformer Block — repeated ×N (80 in Llama 3 70B)"]
+ direction TB
+ SA["`**Self-Attention**
+tokens mix info across the sequence
+q · k · v · o proj`"] --> FF["`**Feed-Forward Network**
+each token processed on its own
+gate · up · down proj`"]
+ end
+ BLK --> H["`**LM Head**
+score every possible next token`"]
+ H --> NT["`**next token** → 'fox'`"]
+ NT -. "append it, run the whole stack again — autoregressive" .-> E
+
+ classDef io fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#0b1f44;
+ classDef emb fill:#ede9fe,stroke:#7c3aed,stroke-width:1.5px,color:#2a1257;
+ classDef attn fill:#fff7ed,stroke:#ea580c,stroke-width:1.5px,color:#431407;
+ classDef ffn fill:#ecfdf5,stroke:#059669,stroke-width:1.5px,color:#052e16;
+ classDef head fill:#fee2e2,stroke:#dc2626,stroke-width:1.5px,color:#450a0a;
+ class P,NT io;
+ class E emb;
+ class SA attn;
+ class FF ffn;
+ class H head;
+ style BLK fill:transparent,stroke:#94a3b8,stroke-dasharray:4 4,color:#475569;
+*/}
+
+
+
+“Layer 1, Layer 2, … Layer N” just means _this identical block, stacked N times_ — 80 times for Llama 3 70B. And each block has exactly two parts:
+
+
+{/* mermaid
+flowchart TB
+ IN["`token vectors **in**`"] --> Q & K & V
+ subgraph SA["Self-Attention · mix information ACROSS tokens"]
+ direction TB
+ Q["q_proj"] --> AT{{"scaled dot-product attention · softmax"}}
+ K["k_proj"] --> AT
+ V["v_proj"] --> AT
+ AT --> O["o_proj"]
+ end
+ O --> G & U
+ subgraph FF["Feed-Forward Network · process EACH token alone"]
+ direction TB
+ G["gate_proj"] --> X(("⊗"))
+ U["up_proj"] --> X
+ X --> D["down_proj"]
+ end
+ D --> OUT["`token vectors **out** → next block`"]
+
+ classDef io fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#0b1f44;
+ classDef lin fill:#f8fafc,stroke:#475569,stroke-width:1.2px,color:#0f172a;
+ classDef op fill:#fef9c3,stroke:#ca8a04,stroke-width:1.2px,color:#422006;
+ class IN,OUT io;
+ class Q,K,V,O,G,U,D lin;
+ class AT,X op;
+ style SA fill:#fff7ed,stroke:#ea580c,color:#431407;
+ style FF fill:#ecfdf5,stroke:#059669,color:#052e16;
+*/}
+
+
+
+Hold onto that one-line distinction, because it’s the soul of the architecture:
+
+> **Self-attention** lets tokens _talk to each other_ (a word looks at its context). The **feed-forward network** then lets each token _think on its own_. Mix, then think. Repeated N times.
+
+And here’s the reassuring part: every box with `_proj` in its name is **just a linear layer — a weight matrix**, exactly the kind you counted in the last post. Self-attention is 4 of them; the FFN is 3. That’s _all_ the learned weights are. The famous “12 × d² parameters per layer” is literally these seven matrices added up (≈4·d² for attention + ≈8·d² for the FFN). Nothing new — just assembly.
+
+So the only genuinely new idea to understand is self-attention. Let’s earn it.
+
+## Why attention has to exist
+
+A word’s meaning depends on its neighbors. “fox” in _“the quick brown fox”_ is an animal; “fox” in _“Fox News anchor”_ is a brand. Same token, same starting embedding — different meaning, and the _only_ difference is the surrounding words.
+
+So a token’s raw embedding can’t be its final representation. Each token needs a way to **look at the other tokens and pull in the relevant ones** to sharpen its own meaning. That act of looking-and-pulling is attention. The question is _how_ a token decides which others are relevant — and that’s what Q, K, and V are for.
+
+## Q, K, V — what they actually mean
+
+This is the part that never clicks from formulas alone, so here’s the analogy that makes it click, then the concrete version.
+
+**Think of a library search.** You walk in with a question. Every book has a short label on its spine advertising its topic. You compare your question against every spine label to judge how relevant each book is, then you actually read the contents of the books — paying most attention to the most relevant ones — and synthesize an answer.
+
+That’s attention, exactly:
+
+| Role | Library | In attention | The token is saying… |
+| --- | --- | --- | --- |
+| **Query (Q)** | your question | what _this_ token wants to know | _“I’m ‘fox’ — what around me tells me which fox I am?”_ |
+| **Key (K)** | each book’s spine label | what each token _advertises_ about itself | _“I’m ‘brown’ — I’m a color/description.”_ |
+| **Value (V)** | each book’s actual contents | the information a token hands over if attended to | _“here’s the actual meaning I contribute.”_ |
+
+Every token produces all three, by passing its own vector through three different weight matrices — `q_proj`, `k_proj`, `v_proj`. Same input vector, three different learned “lenses”:
+
+
+
+
+
+## Attention in three steps (with real numbers)
+
+Now watch one token — “fox” — update itself by attending over the four tokens “The quick brown fox”. “fox” is the 4th token, so it has query `Q4`.
+
+**Step 1 — score.** Compare `Q4` against every token’s Key using a dot product. A dot product is just a similarity number: high when two vectors point the same way. This asks _“how relevant is each token to what fox wants to know?”_
+
+```text
+ Q4 · K1 → 0.2 ("The" — barely relevant)
+ Q4 · K2 → 3.1 ("quick" — very relevant)
+ Q4 · K3 → 1.4 ("brown" — somewhat)
+ Q4 · K4 → 2.7 ("fox" — itself, relevant)
+```
+
+**Step 2 — normalize with softmax.** Raw scores aren’t weights; softmax turns them into fractions that sum to 1 (and exaggerates the big ones):
+
+```text
+ raw: [0.2, 3.1, 1.4, 2.7]
+ softmax: [.03, .52, .10, .35] ← attention weights, add to 1.0
+```
+
+So “fox” decides: pay **52%** attention to “quick”, **35%** to itself, **10%** to “brown”, **3%** to “The”.
+
+**Step 3 — blend the Values.** Take each token’s Value vector, scale it by that weight, and add them up. Then one more matrix, `o_proj`, mixes the result — and that’s the new, context-aware vector for “fox”:
+
+```text
+ new_fox = .03·V1 + .52·V2 + .10·V3 + .35·V4 → o_proj → updated "fox"
+```
+
+
+
+
+
+That’s it — that’s attention. A weighted average of everyone’s Values, where the weights come from how well your Query matches their Keys. Do this for every token, in every block, and information flows across the sentence until each token’s vector encodes not just itself but its whole relevant context.
+
+Don’t take my word for the numbers — play with them. Drag the Query below, or click a token to make the Query mimic its Key, and watch the scores and weights recompute live:
+
+
+
+
+
+Real models run attention several times in parallel — **multi-head attention**. Each _head_ gets its own slice of the vector and its own Q/K/V, so different heads can specialize: one tracks grammar, another long-range topic, another position. Llama 3 70B has **64 query heads** of dimension 128 (64 × 128 = 8192). But it deliberately shares Keys and Values across groups of heads — only **8 KV heads** — so there’s less K/V to store. That’s exactly the `8` in the KV-cache formula later, and the trick has a name: **grouped-query attention (GQA)**. It’s your first glimpse of an optimization baked right into the model to shrink the cache.
+
+## The root cause: generating text one token at a time
+
+Here’s where memory enters, and it’s the whole point. An LLM does **not** write a sentence at once. It’s **autoregressive**: it generates _one token, appends it to the input, and runs the entire model again_ to get the next one. Like predictive text that feeds its own output back in, forever.
+
+```text
+ "The quick brown" → model → "fox"
+ "The quick brown fox" → model → "jumps"
+ "The quick brown fox jumps" → model → "over"
+ ...
+```
+
+Now connect that to attention. To generate the next token, its Query must be scored against the Keys and Values of **every token so far**. Token 1000 attends over 999 predecessors. Which raises the obvious, expensive question: at each step, do we recompute K and V for the entire history?
+
+We could — and it would be catastrophic. Recomputing all past K/V every step is O(n²) work over a sequence. So instead we do the one optimization that makes generation viable at all: **we compute each token’s K and V once, and cache them.** New token arrives → compute its K and V → **append to the cache** → done. The Keys and Values of old tokens never change, so there’s no reason to ever recompute them.
+
+
+
+
+
+**And **_**there**_** is your root cause.** The cache grows by one K/V pair **per token, per layer, per head** — and it never shrinks during a request, because every future token might still want to attend to it. That’s the whole answer to “why does memory keep getting bigger over each iteration”:
+
+- **Weights are constant** — 140 GB whether you generate 1 token or 100,000. Loaded once, shared by everyone.
+- **The KV cache is the opposite** — it starts near zero and grows linearly with every token generated, and it’s _private to each request_.
+
+Put real numbers on it for Llama 3 70B: `2 (K,V) × 80 layers × 8 KV-heads × 128 head-dim × 2 bytes ≈ 320 KB per token`. At 32K tokens → **10 GB**, for one conversation. Now you know _why_: it’s 320 KB accumulated 32,768 times, one autoregressive step at a time.
+
+Here’s that formula as a live calculator — pick a model, drag the context length and the number of concurrent users, and watch the KV cache fill the GPU against the fixed weights (it’s the Red Hat slide that started this, except now you can break it yourself):
+
+
+
+
+
+Drag it around and the scaling speaks for itself. Here is the same model at the context lengths people actually deploy:
+
+
+
+| Context | What it is | KV cache (one request) |
+| --- | --- | --- |
+| 2K | one chat turn | 640 MB |
+| 8K | standard | 2\.5 GB |
+| 32K | long document / codebase | 10 GB |
+| 128K | Llama 3’s maximum | 40 GB |
+
+
+
+Sit with that last row. A single 128K-context request needs **40 GB of KV cache — nearly a third of the model’s entire 140 GB of weights — for one user’s conversation.** Now serve **10** such users at once: **400+ GB of KV cache alone**, on top of the model. That is more than the _entire_ memory of a 4×80 GB node. This — not the math, not the weights — is why managing the KV cache is the single biggest job of a production inference server.
+
+## Where does all this live? The GPU memory hierarchy
+
+We’ve been saying “GPU memory” as if it’s one thing. It isn’t — and its structure is the last piece that makes _why data movement dominates_ finally click. A GPU has three tiers, and they obey the same law as every memory system you already know, from a database’s buffer pool to your CPU’s caches: **the closer to the compute, the faster — and the smaller.**
+
+
+
+
+
+If you know CPU caches, you already know this shape: **SRAM is the L1 cache** — tiny, on-chip, sitting right next to the compute units (the _tensor cores_) and feeding them at a staggering \~19 TB/s. **HBM is main RAM** — gigabytes, but an order of magnitude slower. **Host DRAM is the far-off disk**, reached over a thin \~12 GB/s cable. Now place our two tenants:
+
+- **Model weights (140 GB)** load into **HBM** once at startup and stay there for the server’s life.
+- **The KV cache** also lives in **HBM**, growing per request — which is why _HBM capacity_, not compute, caps how many users you can serve.
+- Every forward pass, small chunks of weights and KV are pulled **HBM → SRAM** for the tensor cores to work on, then results are written back.
+
+And there’s the physical reason decode is _memory-bound_: the tensor cores are so fast they finish and sit idle, **waiting for the next chunk to arrive over the HBM→SRAM path**. The bottleneck isn’t the arithmetic — it’s the data movement.
+
+This also finally explains **why we cache K and V but not Q.** Tensors come in two kinds. **Persistent** ones — the weights and the KV cache — stay in HBM because they’re needed again and again. **Transient** ones — a token’s Q, its freshly-computed K and V, the attention output, the FFN output — are born in SRAM, used immediately, and discarded. Q is transient (used once, the moment its token does the looking); K and V get _promoted_ into the persistent cache precisely because every future token will still need them.
+
+Vocabulary we’ll lean on from here: the umbrella term for any array of numbers — a single value, a vector, a matrix, or higher — is a **tensor**. Weights, Q/K/V, and the KV cache are all tensors. “Data moving through the model” just means tensors moving between these memory tiers.
+
+That leaves one sentence the entire optimization field hangs on: **every technique you’ll meet is one of three moves — move less data, move it more efficiently, or manage the memory better.** Quantization shrinks the bytes; paged KV manages the cache; FlashAttention keeps more of the work inside SRAM. Same three levers, over and over.
+
+## Why this is the hinge of the entire field
+
+Step back and the two costs you now understand explain nearly every inference technique in existence:
+
+- **Memory-bound decode** — each step drags all 140 GB of weights through the chip to compute _one_ token. Barely any math per byte. That’s why decode is slow and why batching many users’ tokens together is the throughput lever. (A dedicated post on this — the roofline model — is coming.)
+- **The growing KV cache** — caps how many users fit in memory at once, and makes long context expensive. That’s why **paged KV cache, quantized KV, GQA, MLA, prefix sharing** all exist. Every one is a strategy to store, shrink, or reuse this cache.
+
+This is why memory dominates every conversation about inference, and the reason is structural: an autoregressive transformer is a machine that re-reads all of its weights and its entire growing memory to produce one token at a time. The whole field of inference optimization is the collected set of tricks for making that bearable.
+
+## What comes next
+
+Everything downstream builds on the two costs in this post — and the highest-leverage trick attacks the bigger one, the KV cache. Under naive allocation, most of a request’s reserved cache sits _unused_: you set aside room for the full context length, but a typical request never fills it. **Paged KV cache** — the idea behind vLLM’s PagedAttention — reclaims that waste with a technique borrowed straight from operating-system memory paging. That’s where this series goes next.