The building blocks of a production LLM app

Property of the model	Consequence	Block that handles it
Stateless: it knows nothing about this user, their data or the conversation	Context has to be fetched and assembled on every call	Orchestrator, context sources, session and memory
Untrusted: inputs can be attacks, outputs can be wrong or unsafe	Nothing it reads or writes can be trusted as-is	Input and output guardrails, permission-scoped tools
Metered and remote: per-token cost, rate limits, outages, latency	Every call costs money and can fail	API layer quotas, LLM gateway (routing, fallbacks, caching, cost)
Probabilistic: same input, different outputs; quality is a spectrum	Unit tests can't prove it works, and changes drift quality	Observability, evals, feedback store, versioned config

Property of the model

Consequence

Block that handles it

Stateless: it knows nothing about this user, their data or the conversation

Context has to be fetched and assembled on every call

Orchestrator, context sources, session and memory

Untrusted: inputs can be attacks, outputs can be wrong or unsafe

Nothing it reads or writes can be trusted as-is

Input and output guardrails, permission-scoped tools

Metered and remote: per-token cost, rate limits, outages, latency

Every call costs money and can fail

API layer quotas, LLM gateway (routing, fallbacks, caching, cost)

Probabilistic: same input, different outputs; quality is a spectrum

Unit tests can't prove it works, and changes drift quality

Observability, evals, feedback store, versioned config

Running example: an internal HR and IT assistant for a 10,000-person company. One request goes through every block: "Can I expense a monitor for home, and open a ticket to order one?"

Client / UI

Why: the user sees latency and trust here, and it's where the system learns.

Stream tokens; show sources next to the answer.
Confirm before any action; thumbs up/down and "report" buttons.

answer streams in with a link to the expense policy, and a "Create ticket?" button

leaving out the UI. Without feedback capture here, the improvement loop never gets data.

API layer

auth · identity · quotas

Why: the model can't know who is asking, so this layer must, and it passes that identity down.

SSO auth, per-user and per-team quotas, request validation, session state.
Attach the user's identity and entitlements to the request context.

user = priya@corp, groups = [eng, india-office]

"Identity is set once at the edge and carried through the whole request, into retrieval and every tool call."

Input guardrails

Why: the input is untrusted.

Redact PII, detect prompt injection and abuse, check the topic is in scope.
Use cheap classifiers and rules here. A big model adds cost and latency.

treating guardrails as one input filter. Output and tool calls need checks too (steps 6 and 8).

Orchestrator

the app's own logic

Why: the model is stateless, so something has to decide what to fetch, build the prompt and parse the result.

Classify intent, then run a fixed workflow where you can and an agent loop only where you must.
Assemble the prompt from a versioned template; parse structured output.

intent = policy question + action → retrieve policy → answer → propose ticket

"I default to a deterministic workflow. An agent loop costs several times more and is harder to test, so it has to earn its place."

Context sources

retrieval · data · tools · memory

Why: the model doesn't know your data. This is where most of the answer's quality comes from.

RAG index (hybrid search + rerank), SQL and APIs, tools that act, user memory.
Filter by the user's permissions at query time.

search only docs Priya's groups can read; ticket tool runs with her token, not a service account

a shared index with no ACL filter. That's how one employee reads another team's documents.

LLM gateway

one door to every model

Why: calls are metered and can fail, so one place handles that for every feature.

Routing (small vs large model), retries, fallback provider, timeouts.
Prompt and response caching, API keys, rate limits, token budgets, cost per team and feature, logging.

putting business logic in the gateway. It knows models, not HR policy.

Models

API or self-hosted

Why: this is the only box that generates text, and it's swappable behind the gateway.

Buy (API) for quality and speed to ship; self-host for data residency, very high volume or fine-tuned small models.

"Because everything goes through the gateway, changing models is a config change plus an eval run, not a rewrite."

Output guardrails

Why: the output is untrusted too.

Validate the schema, check claims against the sources, run policy and PII-leak checks.
Gate actions: the model proposes and code checks before anything runs.

ticket payload validated; user clicks "Create ticket" before it's filed

∞

Cross-cutting

around every hop

Why: the model is probabilistic, so you only know it works by measuring.

Observability: a trace per request with spans for every hop, plus tokens, cost, latency and errors.
Evals: an offline golden set gates every prompt or model change; online metrics in production.
Feedback store: thumbs-down and escalations become new eval cases.
Config registry: versioned prompts, model choices and feature flags, so changes can be rolled back.

logging full prompts with PII in them. Redact before storing traces, and set a retention limit.

The skeleton to draw

Client (stream, cite, feedback) │ ▼ API layer (auth, identity, quotas) ──► Input guardrails │ ▼ ┌──────────────── Orchestrator ────────────────┐ │ (intent → workflow | agent, prompts) │ ▼ ▼ Context sources (ACL-filtered) LLM gateway RAG · SQL · APIs · tools · memory (route, fallback, cache, cost) │ ▼ Models (API | self-hosted) ┌───────────────────────────────────────────────┘ ▼ Output guardrails ──► Client Around everything: traces · evals · feedback store · prompt/config registry

Back-of-envelope for the example

Quantity	Estimate	What it decides
Traffic	3,000 daily users × 3 ≈ 9k req/day ≈ 1 QPS peak	Throughput is trivial; the gateway is about cost and reliability
Tokens per request	≈ 4,000 in + 400 out	Input dominates, so keep retrieved context tight
Cost (at $3 / $15 per M tokens)	≈ $0.018 / req → ≈ $3.6k / month	Cheap compared with support staff time
Prompt caching (1.5k stable tokens)	≈ –$0.004 / req → ≈ 20% saved	Put system prompt and tool definitions first
Agent loop instead of workflow	≈ 8 calls → ≈ 6–8× cost	Default to workflows
Trace storage	≈ 15 KB × 200k ≈ 3 GB / month	Storage is cheap; PII redaction and retention are the issue

All figures are illustrative assumptions. Say them out loud as assumptions in an interview.

What breaks, and which block fixes it

Failure	Fix	Block
User sees documents they shouldn't	Carry identity through; ACL filter inside the retrieval query	API layer, context sources
Injected text in a document triggers a tool	Treat context as data; user-scoped tools; confirm actions	Orchestrator, output guardrails
Provider outage or rate limit	Fallback model or provider; degrade to search results	LLM gateway
Cost spike from loops or abuse	Step limit, token budget per request, quotas	Gateway, API layer
Quality drops after a prompt edit	Versioned prompts; eval gate before rollout	Config registry, evals
Answers cite outdated policy	Incremental ingestion with a freshness target	Context sources

The 30-second answer

"An LLM is a stateless, untrusted, metered, probabilistic function, so a production app is mostly the system around it. On the request path: a client that streams and collects feedback; an API layer for auth, quotas and the user's identity; input guardrails; an orchestrator that runs a workflow, or an agent only when needed, and assembles context from retrieval, databases, tools and memory, filtered by that user's permissions; an LLM gateway for routing, fallbacks, caching and cost; the models; then output guardrails that check schema, grounding and policy before anything is shown or done. Around all of it: tracing, evals, a feedback store and a versioned prompt registry. Then I say which two or three boxes matter most for this problem, and go deep on those."