Streaming, sync, async or batch: choosing how an LLM feature runs

Who is waiting	How long they can wait	Mode
A person, reading as it is written	First token in about 1 s, full answer can take 10 s or more	Streaming
Code, before its next step	A few seconds	Sync request/response
A person, who can come back later	Tens of seconds to hours	Async job
No one	Hours, often up to 24 h	Batch

Who is waiting

How long they can wait

Mode

A person, reading as it is written

First token in about 1 s, full answer can take 10 s or more

Streaming

Code, before its next step

A few seconds

Sync request/response

A person, who can come back later

Tens of seconds to hours

Async job

No one

Hours, often up to 24 h

Batch

Streaming

interactive · perceived speed

Example: the agent-assist chat that drafts a reply while the support agent watches.

Send tokens as they are generated over SSE or WebSockets. The full answer still takes seconds, but the user starts reading in under one.
Measure time to first token (TTFT) and tokens per second, not total latency.
Handle cancellation: when the user hits stop or closes the tab, abort the model call so you stop paying for tokens no one will read.
Run output guardrails on chunks or sentences, and have a way to retract text that fails a check.

"For chat I stream, because what users feel is time to first token. A 12-second answer that starts in 600 ms feels fast."

streaming JSON that code has to parse. Partial JSON is not valid JSON. Stream to people, not to parsers.

Sync request/response

short · needed before the next step

Example: classify each incoming ticket's intent and extract the order ID as JSON, then route it.

Short, structured output that code consumes whole. Nothing useful to show halfway.
Use small, fast models and keep outputs short: output tokens are the slow part.
Set a timeout and a retry budget, and have a fallback (a rule, a default queue) when the call fails.
Validate against a schema before the next step uses it.

"Intent classification sits in the request path, so it's a sync call to a small model with a 3-second timeout and a default route if it fails."

a sync call with no timeout. One slow provider response holds a thread and can cascade through the workflow.

Async job

seconds to hours

Example: "Summarise this customer's last 200 tickets into an account report." An agent reads, calls tools and writes for several minutes.

Accept the job, return a job ID at once, and do the work in a worker that reads from a queue.
Give a status API and a webhook or push notification when it's done.
Checkpoint long work, so a crashed worker resumes from step 7 instead of step 1.
Show progress: "Reading 200 tickets… Grouping issues… Writing summary".

"Anything that can run past the gateway timeout becomes a job: 202 with a job ID, a queue, workers with checkpoints, then a status endpoint and a webhook."

holding an HTTP connection open for minutes. It works in dev, then dies at the load balancer in production.

Batch

hours acceptable

Example: every night, tag all of the day's tickets by topic and sentiment for the analytics dashboard. Or backfill embeddings for the whole knowledge base.

Use the provider's batch API: upload a file of requests, collect results within a window (often 24 h), usually at about 50% off.
Batch traffic doesn't use your real-time rate limits, so it doesn't compete with live users.
Self-hosted: run on spare or spot GPUs with big batch sizes, optimising for throughput, not latency.
Make each item retryable on its own, and keep a record of which IDs are done.

10M items × 500 tokens = 5B tokens · at $1 per 1M ≈ $5,000 live, ≈ $2,500 batch (illustrative)

"If no one is waiting, I use the batch API. It's about half the price and stays out of my live rate limits."

looping over 10 million items with the real-time API. You hit rate limits, pay full price and slow down live traffic.

The four modes side by side

Mode	Latency need	Examples	Notes
Streaming	Interactive, perceived speed	Chat, drafting emails, writing help	SSE or WebSockets; measure TTFT; handle cancellation
Sync	Short, needed before the next step	Intent classification, JSON extraction in a workflow	Timeouts and fallbacks; small fast models; schema validation
Async job	Seconds to hours	Report generation, video summaries, agent tasks, deep research	Queue, workers, checkpoints, status API, webhook or notification
Batch	Hours acceptable	Tag 10M products, nightly summaries, embeddings backfill	Batch APIs (often ~50% off), spot GPUs, maximum throughput

The async job pattern

Client ──POST /jobs (Idempotency-Key: abc123)──► API │ store job: queued ◄──202 Accepted {job_id}──────────────────┤ ▼ Queue │ ▼ Worker: process with checkpoints │ store result: done ▼ Client ──GET /jobs/{id}──► {status, progress, result_url} or ◄── webhook / push notification when done

Use polling for simple clients and a webhook or push notification when the caller is a server or a mobile app. Offer both if you can.

What breaks, and the fix

Concern	What goes wrong	Fix
Timeouts	Gateways and load balancers cut connections at 30 to 60 s	Anything that can run longer is an async job
Idempotency	A client retries the submit and the job runs twice, at twice the cost	Idempotency key on submit; return the existing job ID for a repeat
Progress UX	A spinner for 3 minutes looks broken, so users refresh and resubmit	Stream intermediate steps: "Reading 12 documents…"
Priority	A nightly batch takes all the model capacity and live chat slows down	Separate queues or capacity; interactive first; batch API for bulk
Cancellation	Abandoned streams and jobs keep burning tokens	Abort the model call on disconnect; a cancel endpoint for jobs

The 30-second answer

"I choose by who is waiting and how long they can wait. If a person is reading as it's written, like chat or drafting, I stream over SSE so the first token arrives in under a second, and I handle cancellation. If code needs a short, structured result before its next step, like intent classification or JSON extraction, I make a sync call to a small fast model with a timeout and a fallback. If the task can run longer than the 30 to 60 second gateway timeout, like reports or agent tasks, it becomes an async job: 202 with a job ID, a queue, workers with checkpoints, a status API and a webhook, with idempotency keys and progress updates. If no one is waiting, like tagging millions of products overnight, I use batch APIs, which are often about half the price, and keep them on separate capacity so they never slow down live users."