Turn users into requests, requests into tokens, and tokens into dollars, GPUs, storage and latency. Use round numbers and say every assumption out loud. The interviewer is grading the chain of reasoning and what you decide from it, not whether you land on the exact figure.
▶ Watch the 4:52 explainer
Two requests can differ 100x in cost. So every estimate runs through one chain: users → requests → tokens → cost and capacity. Get the token count per request right and everything else is multiplication.
| Classic system estimate | LLM system estimate | Why it changes |
|---|---|---|
| Requests per second | Tokens per second (input and output separately) | Rate limits and GPU throughput are measured in tokens |
| Bytes per request | Tokens per request, which grows with history and retrieval | A chat turn re-sends the whole conversation |
| Cost per server hour | Cost per request = tokens × price | Cost scales with usage and prompt design, not just traffic |
| Response time in ms | Time to first token + output tokens ÷ decode speed | Long answers take seconds, so requests stay open |
Running example: a chat assistant with 1M daily active users. All prices are illustrative.
Why: everything downstream multiplies off this number, so state it first.
"I'll size for peak, about 700 requests per second, because rate limits and capacity bite at peak, not on average."
sizing for the average. Traffic follows time zones and work hours, so a 2–5x peak factor is normal.
Why: this is the number that decides cost. Break input into its parts so you can pull levers on each.
using one turn's size. History is re-sent every turn, so turn 4 carries turns 1–3. Use the average over the whole conversation.
Why: cost per request is the number product and finance care about. Illustrative prices: $1 per 1M input, $5 per 1M output.
"About a third of a cent per turn, two million a month. Input is 87% of the tokens and the larger share of cost, so caching and shorter context are my first levers."
assuming input dominates cost because it dominates volume. Output costs 5x more per token here, so it is still 43% of the bill.
Why: the provider limits you in tokens per minute, and self-hosted GPUs are sized in tokens per second.
"Eighty-four million input tokens a minute at peak is beyond a default API tier, so I'd plan for a negotiated limit, several regions, or self-hosting part of the traffic."
checking only output tokens. Input-token limits are usually hit first because input is 7x larger.
Why: logs grow fast, and long outputs keep connections open.
"Two hundred gigabytes of logs a day means I'd set a retention window and sample full traces rather than keep everything."
quoting total latency only. Budget time to first token (network + retrieval + prefill) separately from decode time.
1M users × 20 turns = 20M requests/day 20M ÷ 86,400 ≈ 230 QPS → peak ×3 ≈ 700 QPS × 2,000 in + 300 out tokens = 40B in + 6B out per day × $1/M in, $5/M out = $40K + $30K = $70K/day ≈ $2.1M/month ÷ 20M ≈ $0.0035 per turn 700 QPS × 300 out = 210K output tokens/s at peak 20M × 10 KB logs = 200 GB/day
| The number says | So you decide | Rough effect |
|---|---|---|
| Input is 87% of tokens | Prompt caching for the system prompt and history, trim history, retrieve fewer chunks | input bill down 30–60% |
| Most turns are easy | Route 60% of turns to a model 10x cheaper | 0.4 + 0.6 × 0.1 = 46% of cost |
| 84M input tokens/min at peak | Negotiate limits, spread across regions, or self-host part of the load | avoid 429s at peak |
| 6 s to finish an answer | Stream tokens, cap output length, keep about 4,200 connections open | first token < 1 s |
| 200 GB of logs a day | Retention window, sample full traces, keep metrics for everything | 6 TB/mo → far less |
Routing check: 60% of turns at a tenth of the price leaves 40% + 6% = 46% of the bill, about $32K a day instead of $70K. That is "more than half" saved.
"In an LLM system the unit of cost is the token, so I run one chain. Users times actions gives requests per day: a million users at 20 turns is 20 million, about 230 QPS on average and 700 at peak. Times tokens per request, counting history and retrieved context on the input side: 2,000 in and 300 out gives 40 billion input and 6 billion output tokens a day. Times price: about $70K a day, roughly a third of a cent per turn. Then capacity at peak: 210K output tokens per second, so I check rate limits or GPU count. I add storage, about 200 GB of logs a day, and a latency budget. Then I say what the numbers mean: cache and trim the input, and route easy turns to a cheaper model."
The video is narrated by an AI voice and was scripted with AI assistance. Watch it on YouTube.