Netflix encodes millions of videos in batch mode overnight. ChatGPT uses streaming to show you tokens as they're generated. Your bank's fraud detection uses synchronous REST — it must respond before your transaction completes. Same ML model, three completely different serving architectures.
Which architecture would you choose for a medical diagnosis tool that must respond in under 200ms?
| Mode | Latency | Throughput | Best for |
|---|---|---|---|
| REST | Low (~120ms) | Medium | Real-time inference |
| Batch | High (~500ms+) | High | Offline processing |
| Streaming | TTFT = Low | Medium | Generative AI UX |