arthrop0d

Brief articles on varied topics.

LLM Key-Value Caches During Autoregressive Generation

artificial intelligenceperformance

A key-value (KV) cache stores the attention keys and values already computed for each earlier token, at every transformer layer, while a language model generates text. On the next decoding step, the model reuses those tensors instead of calculating them again.

The problem: every new token has the whole history

An autoregressive language model generates one token at a time. A token is a text fragment represented internally by a numeric vector. To choose the next token, the model processes the prompt and everything it has generated so far.

The relevant operation is self-attention: each token compares its current information with earlier tokens and uses the best matches to gather information from them. For each token, a transformer layer produces three vectors:

The model compares the current query with the keys from previous positions, turns those match scores into weights, and combines the corresponding values.

Without a cache, generating token 100 means running the model over tokens 1 through 99 again, even though their representations have not changed. Generating token 101 repeats that work for tokens 1 through 100. The output is correct, but the same prefix is repeatedly processed.

The KV cache does not remember the generated text as text. It remembers intermediate numeric tensors needed by later attention operations.

What the cache contains

For each transformer layer, the cache holds two collections:

They are usually arranged by batch item, position, attention head, and the size of each head's vector. An attention head is one independent set of query, key, and value projections inside a layer; multiple heads let the model examine different relationships at once.

The cache is normally specific to one active sequence. It includes the prompt's positions and the tokens generated so far. It does not usually include the query vectors, because an old query is not needed when a new token arrives. It also does not store every hidden activation from the forward pass: the model still computes the new token's hidden state and query at each layer.

How one decoding step uses it

There are two phases worth separating:

  1. Prefill processes the initial prompt. The model computes keys and values for all prompt positions and places them in the cache. This phase can process many tokens in parallel.
  2. Decode generates tokens one at a time. For each new token, the model computes that token's layer inputs, query, key, and value. It appends the new key and value to the cache.

At a layer during decoding, the new query attends to the concatenation of the cached keys and the new key. The resulting attention operation reads from the cached values and the new value. The model then continues through the remaining computations in that layer and passes the new token's result to the next layer.

Conceptually, the operation looks like this:

# Conceptual shape: one new token attends to all positions so far.
keys = concatenate(cached_keys, new_key)
values = concatenate(cached_values, new_value)
output = attention(new_query, keys, values)

cached_keys = keys
cached_values = values

This is a description of the data flow, not a prescription for a particular framework API. Production implementations often avoid physically copying the entire cache when appending.

With caching, old keys and values are projected once and then reused. The model still has to compare the new query with every previous key, so attention for one new token still grows with the context length. The important saving is that it no longer recomputes the old tokens' keys, values, and other per-token layer work at every step.

The speedup and its cost

For a context of length L, an uncached implementation repeatedly processes the growing prefix. A cached implementation processes the new token once per layer and performs an attention read across the existing L positions. This turns decoding from repeatedly rebuilding the prefix into an incremental operation whose per-token attention work grows roughly linearly with context length.

The trade-off is memory. A rough KV-cache size is:

2 × layers × positions × KV heads × head dimension × bytes per element

The first factor of two accounts for keys and values. The size is multiplied by the number of concurrent sequences in a batch. For example, a model with 32 layers, 32 key/value heads, head dimension 128, 4,096 positions, and two-byte values needs about 2 GiB for the cache alone. Model weights, temporary buffers, and framework overhead require additional memory.

Some models use grouped-query attention (GQA) or multi-query attention (MQA). These arrangements use fewer key/value heads than query heads, so they reduce KV-cache memory while retaining more query heads for computation. Cache quantization can reduce it further by storing values with fewer bits, usually with a possible quality or performance trade-off.

Where the cache appears in practice

The feature may appear in configuration as use_cache, or in interfaces as past_key_values. Logs may distinguish prompt processing from token generation, because the first fills the cache and the second consumes it. A request that has a long prompt, a large maximum output, or many concurrent users can exhaust GPU memory even when the model weights fit comfortably.

A KV cache therefore explains a common serving pattern: generation becomes much faster than recomputing the entire conversation for every token, but memory usage rises with context length and active requests. It is a deliberate exchange of memory for decoding speed.

← All articles