Serving LLMs is a product decision

The fastest model in a benchmark is not necessarily the model I would put in front of a user. The benchmark may describe an isolated request; the product has real traffic, uneven prompts, failures, budgets, and people waiting for a reply. Choosing how to serve a large language model (LLM) means deciding which of those constraints the product can live with.

This is why I think of LLM inference and model serving as product decisions. The infrastructure matters, but it is in service of an experience.

Start with the moment the user feels

“Latency” can hide several different experiences. A conversational product may feel slow when it takes too long to show the first word, even if the complete response arrives quickly afterward. A background task may care far more about total completion time. A long answer that streams smoothly can feel different from one that pauses unpredictably. The right measurement depends on what the user is doing.

I would start by describing one interaction in plain language. What is the user trying to accomplish? How long can they reasonably wait before seeing progress? What happens if the model fails halfway through? Is a lower-quality answer worse than a delayed one? Only then would I choose the measurements that represent that experience.

For a streaming response, time to first token (TTFT) and inter-token latency are useful lenses. End-to-end latency still matters, but a single average can conceal an uneven response or the requests that wait longest. I would look at p95 latency under a workload that resembles the product, including its mix of prompt and output lengths. A clean number from an empty server is a starting point, not a deployment decision.

Self-hosted LLM inference or an API?

The apparent choice is often “host the model ourselves or call an API.” In practice, each option changes what the team must own.

With an API, the team buys a service boundary. That can make it easier to try a model, scale without managing GPUs, and move quickly. The team still owns integration, quality evaluation, fallback behavior, spending limits, and what happens when a provider changes or fails.

With self-hosting, the team gets more direct control over the serving path. Engines such as vLLM or SGLang are options for that path, but choosing an engine does not settle capacity, deployment, monitoring, or recovery. GPU utilization, batching, and memory used by the key-value (KV) cache all matter under concurrent load. A GPU that is cheap per token when busy may be expensive when traffic is quiet. A provider that looks expensive per token may be the simpler choice for an unpredictable or small workload.

Neither cost comparison is complete without the shape of demand. How many requests arrive together? How long are the inputs and outputs? Does demand spike at particular times? How much spare capacity is needed to keep the experience stable? The answer can change as the product grows, so the architecture should make it possible to revisit the decision.

Put quality in the same table

Serving discussions can become optimization exercises around token throughput and latency while quietly treating model quality as fixed. It rarely is. A faster or cheaper option that changes the answers people care about may be a worse product choice. A larger model that improves a benchmark but adds delay may be a worse one too.

I would compare candidate paths on the same representative task set. The table would include answer quality, first-response delay, total time, failure behavior, and the cost of operating the path. It would also record what the team does not know yet. This keeps an impressive serving number from settling a question it cannot answer.

There is no universal winner. A product may use one path for an interactive turn and another for background work. It may route only some requests to a more capable model. These are hypotheses to test against user needs, rather than rules to adopt because they sound sophisticated.

Decide, then watch for the decision to expire

A good serving decision has a reason and a review point. I would write down the workload it was tested against, the constraint that mattered most, the alternatives considered, and what signal would make us reconsider. Changes in traffic, model capability, provider terms, or user behavior can all invalidate yesterday's answer.

Speed only matters in relation to the experience it enables. The best serving path is the one a team can explain, measure, operate, and change when the product asks for something different.