Caching in LLM systems, layer by layer

There is no single "LLM cache". There are five, each skipping a different piece of work: the whole call, the input tokens, the embedding, or the database lookup. A good answer names them in the order a request meets them, says what each saves, and says how each one can serve a wrong or leaked answer.

What makes a hit wrong	Which layer it hurts	The defence
Different meaning that looks similar	Semantic cache	A strict threshold and an eval set of near-misses
Different user, tenant or permission	Response and retrieval caches	Scope in the cache key
New model or new prompt	Response and embedding caches	Version in the key
Data changed since it was stored	Retrieval, tool and response caches	TTLs and event-based invalidation

What makes a hit wrong

Which layer it hurts

The defence

Different meaning that looks similar

Semantic cache

A strict threshold and an eval set of near-misses

Different user, tenant or permission

Response and retrieval caches

Scope in the cache key

New model or new prompt

Response and embedding caches

Version in the key

Data changed since it was stored

Retrieval, tool and response caches

TTLs and event-based invalidation

What a request goes through

Request │ normalise text, build scoped key (tenant, permissions, locale, model + prompt version) ▼ Exact cache hit? ── yes ──► return cached response │ no ▼ Semantic cache: similar query ≥ threshold, same scope? ── yes ──► return cached answer │ no ▼ Retrieval / tool result cache (embedding cache sits under both lookups) │ ▼ LLM call with prompt prefix caching (static prefix first, user turn last) │ ▼ Store in caches with TTL + scope keys ──► Response

Two misses in a row are what send a request to the expensive path. Everything after that still uses caches, just smaller ones.

Running example: a customer-support assistant for an e-commerce company, with a 3,000-token system prompt and tool list, and user questions like "how do I return shoes?"

Normalise and scope the key

before any lookup

Why: the key decides both the hit rate and the safety of every response cache.

Normalise: lowercase, trim whitespace, strip punctuation, so "How do I return shoes?" and "how do i return shoes" match.
Scope: tenant, user permissions or role, locale, model version, prompt version.
Key = hash(scope + normalised input).

"The cache key includes tenant, permissions, locale, model version and prompt version. I never serve user A's personalised answer to user B."

keying on the question text alone. That is how one customer sees another customer's order status.

Exact response cache

saves the whole call

Why: the cheapest possible hit. Same normalised input in the same scope returns the stored output in about a millisecond.

Works well for FAQs, buttons and suggested prompts, and repeated workflow steps.
Hit rate on free text is low, because people phrase things differently.
Usually a key-value store such as Redis, with a TTL.

expecting a high hit rate on open chat. Measure it before you promise savings.

Semantic cache

saves the whole call

Why: catches paraphrases. Embed the query, find the nearest stored query in the same scope, and reuse its answer if the similarity is above a threshold.

The threshold is the whole game. Too low gives wrong answers, too high gives no hits.
"Cancel my order" and "cancel my subscription" can sit very close in embedding space and need different answers.
Tune the threshold on a labelled set of near-miss pairs, not by eye.
"Adapting" a cached answer with a small LLM call is possible, but it gives back some of the saving.

"Similar is not the same as equivalent, so I tune the threshold against near-miss pairs and only use the semantic cache for low-risk questions."

one global threshold for every intent. Questions with entities in them, like order IDs and dates, should skip it.

Embedding, retrieval and tool caches

saves sub-calls

Why: on a miss, the request still embeds text, searches an index and calls APIs. Each of those can be cached on its own.

Embedding cache: key = hash(text + embedding model version). Invalidate everything when the embedding model changes, because old and new vectors are not comparable.
Retrieval cache: query plus filters gives the top-k chunk IDs. Invalidate when the index is rebuilt or the documents change.
Tool results: "return policy for shoes" can live for hours. "Order status" lives for seconds or not at all.

one TTL for everything. Set the TTL from how fast that data really changes.

Prompt prefix caching

saves input cost and time to first token

Why: the model reuses the computed attention state (the KV cache) for a prompt prefix it has already processed, so it skips that work. Providers do this server-side, and vLLM does it with automatic prefix caching.

Only an identical prefix hits. One changed token early in the prompt invalidates everything after it.
Put static content first (system prompt, tool definitions, examples, shared documents) and the variable parts last (user message, timestamps, user name).
Provider caches are short-lived, usually minutes, and often need a minimum prefix length. Check your provider's docs.
This one doesn't return stored answers, so it can't serve a wrong answer. It only makes the call cheaper and faster.

"Most of my input tokens are the same system prompt and tools on every call, so I put them first and let prefix caching discount them."

a timestamp or the user's name at the top of the system prompt. It makes every prefix unique and the hit rate drops to zero.

Store, expire and invalidate

after the call

Why: a cache without invalidation rules is a bug you haven't noticed yet.

Store the response with its scope key and a TTL that fits the content.
Don't store refusals, errors or answers that failed a guardrail check.
Invalidate on events: a policy document is edited, a model or prompt version ships, the index is rebuilt.
Monitor hit rate per layer and sample cached answers for quality, just like fresh ones.

The table to draw

Cache	Saves	Risk / caveat
Exact response	Full LLM call	Low hit rate for free text; good for FAQs and repeated workflows
Semantic	Full LLM call	Wrong answer if "similar" isn't equivalent ("cancel order" vs "cancel subscription"); tune the threshold
Prompt / prefix (provider or vLLM)	Input token cost + time to first token	Prefix must be identical; put static content first
Embedding	Embedding calls	Invalidate when the embedding model changes
Retrieval / tool results	Database and API latency	Staleness; set TTLs per data type

Why prefix caching is usually the first win

Per call: 3,000 static prefix tokens + 200 user tokens = 3,200 input tokens

Cached prefix billed at ~10% (assumed discount): 300 + 200 = 500 billed-equivalent

Input cost drops ~84% on every call that hits, even with zero response-cache hits

Exact cache at a 10% hit rate: saves 10% of calls, and only on repeated questions

Discounts and cache lifetimes differ by provider and change over time. State them as assumptions. The point is the shape: prefix caching helps every call, response caches help only repeats.

The 30-second answer

"Caching in an LLM system happens in layers, and each one skips different work. An exact cache returns the stored answer for the same normalised input. A semantic cache reuses an answer for a similar query above a tuned threshold, which is risky because similar isn't always equivalent. Prompt prefix caching reuses the model's KV cache for an identical prefix, cutting input cost and time to first token, so I put static content first. Under those sit embedding caches and retrieval and tool caches with TTLs. Every response cache key includes tenant, permissions, locale, model version and prompt version, so one user never gets another's answer. And I don't cache personalised, time-sensitive or high-stakes answers without revalidating them."