Batch or streaming inference

Good at

• Throughput: large batches keep accelerators busy and cost per item falls sharply.
• Serving reads in single-digit milliseconds, because it is a key lookup.
• Retries, backfills and reruns — a failed batch is rerun, not lost.
• Providers' batch tiers, which are typically around half the price of synchronous calls.
• Inspecting and correcting outputs before any user sees them.

Costs you

• Results are as stale as the schedule — a daily job means up to a day old.
• Cannot respond to anything that happened since the run, including the current session.
• You compute for every entity, including the vast majority nobody will ask about.
• Storage for the whole prediction space, which for pairs or sequences explodes.
• Cold start for new entities that did not exist when the batch ran.

Good at

• Using everything known at request time, including the last thing the user typed.
• Only computing what is asked for, which is the cheaper side when the space is large.
• Token streaming, which cuts perceived latency dramatically — first token in under a second reads as responsive even when the full answer takes ten.
• Serving inputs that did not exist a moment ago.

Costs you

• Capacity must be sized for peak, and idle capacity is paid for.
• Latency is now a product requirement with an SLA attached.
• Small dynamic batches use the hardware far less efficiently than large offline ones.
• Streaming complicates everything downstream: validation, moderation, retries, and parsing structured output mid-stream.
• A failure is user-visible immediately, with no chance to inspect the output first.