1
Clarify
3–5 min
Why: you can't optimise without an objective, and "good" means something different for every product.
- Users and job: customers or internal agents? Answer only, or take actions like refunds?
- One north-star metric: resolution without a human, CSAT or deflection.
- Cost of being wrong. This sets how strict your guardrails need to be.
- Scope: languages, channels, existing data such as the knowledge base and ticket history.
"Is this customer-facing, and can it take actions or only answer? That decides most of the risk."
firing off 15 questions. Ask the 3–4 that change the design and state assumptions for the rest.
2
Requirements, with numbers
~3 min
Why: constraints you can design against and check later.
- Functional: answer from the knowledge base, look up orders, start returns, hand off to a human with context.
- Quality: ≥ 90% grounded answers, < 2% wrong actions.
- Latency: p95 first token < 1 s, streamed.
- Cost: < $0.05 per conversation, compared with a human ticket.
- Safety and scale: no data leaks between users, injection-resistant, peak traffic known.
"low latency" isn't a requirement. "p95 first token under 1 s" is.
3
Estimates that change decisions
3–5 min
Why: numbers rule designs out. Only do math that changes a decision.
- 2M convos/mo × 5 turns = 10M calls ≈ 4 QPS avg, ~40 peak
- ~3k input + 300 output tokens ≈ $0.0135/call ≈ $135k/mo
- vs 2M tickets × ~$5 human ≈ $10M/mo
- 200k chunks × 1024 dims × 4 B ≈ 800 MB → fits in memory on one node
"Input tokens are about two-thirds of the cost, so caching and tight retrieval are my levers. The index fits in memory, so I don't need a distributed vector database."
prices here are illustrative. Use round numbers and state them as assumptions.
4
High-level architecture
~10 min
Why: get a simple version working end to end before optimising any one piece. Start with RAG plus one LLM call, then add pieces and say why.
opening with "LangChain + vector DB + GPT" or an agent framework before stating the problem.
5
Deep dives on the riskiest parts
15–20 min
Why: you can't go deep on everything. Pick what is riskiest or least certain, and say why you picked it.
- Model: API or self-hosted; route easy turns to a small model and hard ones to a large model; fallback provider.
- Retrieval: chunking, hybrid BM25 + vectors (catches SKUs and order IDs), reranking, freshness, citations.
- Tools: the LLM proposes and deterministic code checks and executes; credentials scoped to the user; confirm before irreversible actions.
- Guardrails: retrieved docs and tool output are data, not instructions; grounding check; escalate when confidence is low.
- Serving: streaming, timeouts, retries, prompt caching, rate limits, a token budget per conversation.
6
Evaluation and monitoring
~5 min · raise it yourself
Why: if outputs are probabilistic, measuring is the only way to know the system works.
- Offline: a golden set of about 500 real tickets; retrieval recall@k measured separately from answer correctness and groundedness; an LLM judge checked against human labels; a red-team set; a regression gate in CI.
- Online: resolution and escalation rate, CSAT, thumbs, p95 latency, cost per conversation, guardrail trigger rate, sampled human review.
- Flywheel: escalations and thumbs-down become new eval cases and fixes.
"I score retrieval and generation separately, so when quality drops I know which half broke."
7
Trade-offs, failure modes, rollout
~5 min
Why: senior engineers are judged on what they'd give up and how the system fails.
- Quality ↔ latency ↔ cost: a bigger model is better but slower and costs more, so route by difficulty.
- Rollout: shadow mode → internal staff → 1% of traffic → ramp, with a kill switch and an A/B test against human agents.
