Model Serving in Production: LLM Batching, Scaling & GPUs

Model Serving in Production: Batching, Autoscaling, GPU Efficiency, and LLM Inference

Training a model is only half the system.

The other half is serving it reliably under real traffic.

A production inference service has to answer questions that a notebook never sees:

How many requests can one GPU handle?
What happens when traffic spikes?
Should requests be batched?
How do you avoid wasting KV cache?
How do you roll out a new model without breaking clients?
How do you measure queue time separately from model execution time?
What happens when one model version is slower than another?
How do you serve a model that does not fit on one accelerator?

Model serving is therefore not just “put a REST API in front of a model.” It is a scheduling, resource-management, reliability, and observability problem.

What Model Serving Actually Includes

A minimal inference path looks like:

Client
  ↓
API / inference protocol
  ↓
request queue
  ↓
scheduler / batcher
  ↓
model runtime
  ↓
CPU / GPU / accelerator
  ↓
response

A mature production system adds:

routing
authentication
rate limits
batching
autoscaling
model versioning
health checks
metrics
tracing
rollouts
fallbacks

That is why model serving should be treated as infrastructure, not as a helper function inside your backend.

Separate the Model Runtime From the Serving Platform

Two layers are often confused.

Runtime layer

The runtime is responsible for executing the model efficiently.

Examples include:

vLLM for LLM inference
NVIDIA Triton Inference Server for multi-framework inference
framework-specific runtimes such as TensorRT, ONNX Runtime, PyTorch, or OpenVINO

Serving/platform layer

The platform manages lifecycle and traffic around runtimes.

Examples include:

Kubernetes
KServe
custom inference gateways
managed cloud inference services

A useful mental model is:

KServe / Kubernetes / gateway
          ↓
serving runtime
          ↓
model execution engine
          ↓
hardware

Do not expect one layer to solve every problem.

Throughput vs Latency Is the Central Trade-Off

Inference systems usually optimize for two competing goals:

latency — how long one request waits
throughput — how much work the system completes per unit of time

The configuration that gives the lowest single-request latency may waste accelerator capacity.

The configuration that maximizes throughput may force users to wait in a queue while batches form.

Production tuning should therefore start with explicit service objectives such as:

p95 time-to-first-token &lt; target
p99 end-to-end latency &lt; target
minimum requests/tokens per second
maximum acceptable queue depth

Without latency budgets, batching and autoscaling cannot be tuned rationally.

Dynamic Batching for Traditional Inference

For stateless models, multiple independent requests can often be combined into one accelerator batch.

NVIDIA Triton supports dynamic batching, where requests arriving close together are grouped before execution.

Conceptually:

request A ─┐
request B ─┼──&gt; batch of 4 ──&gt; GPU
request C ─┤
request D ─┘

The benefit is higher throughput because accelerators generally execute larger batches more efficiently than many tiny independent calls.

But waiting for a batch adds queue latency.

Triton exposes controls such as:

maximum batch size
preferred batch sizes
maximum queue delay
queue limits
priorities
request timeouts

The correct batch settings depend on your latency budget and traffic pattern.

Continuous Batching for LLMs

Generative models are different from ordinary fixed-shape inference.

One request may generate 20 tokens while another generates 2,000. If you force a static batch to remain together until every sequence finishes, short requests waste time waiting for long ones.

Modern LLM runtimes such as vLLM use continuous batching.

Instead of treating a batch as permanent:

finished sequence leaves
        ↓
new sequence can enter
        ↓
GPU stays busy

This improves utilization under mixed-length generation workloads.

For LLM serving, scheduler behavior is often as important as raw model FLOPS.

KV Cache Is a First-Class Resource

Autoregressive models repeatedly reuse attention keys and values from previously processed tokens.

Those tensors are kept in the KV cache.

For long contexts and high concurrency, KV cache can consume a major share of GPU memory.

This changes capacity planning.

Your GPU memory is not simply:

model weights

It is closer to:

model weights
+ KV cache
+ temporary activations/workspaces
+ runtime overhead
+ fragmentation/headroom

A model that fits comfortably for one request may fail badly under many simultaneous long-context requests.

Monitor KV cache usage and queue pressure rather than sizing purely from model file size.

Prefix Caching Can Avoid Repeated Prompt Work

Many applications repeatedly send the same prefix:

system prompt
+ tool definitions
+ company policy/context
+ user-specific stable context
+ new request

If hundreds of requests share a large prefix, recomputing its attention state every time wastes compute.

Runtimes such as vLLM support automatic prefix caching so repeated prefixes can reuse computed state where applicable.

This can be especially useful for:

agents with large tool schemas
chat systems with stable system prompts
repeated document prefixes
multi-turn sessions

But caching effectiveness depends on actual prefix reuse. Measure cache hit rate instead of assuming it will help every workload.

Speculative Decoding Targets Decode Latency

Autoregressive generation traditionally produces tokens sequentially.

Speculative decoding uses a cheaper proposal method or model to suggest future tokens, while the target model verifies them.

When enough proposed tokens are accepted, multiple tokens can effectively be advanced with less expensive target-model work.

Current vLLM supports multiple speculative-decoding strategies.

This is not a guaranteed speedup for every model or traffic shape. Benchmark:

output length
acceptance rate
model pair
hardware
batch size
latency percentile

A benchmark that improves tokens/sec but hurts time-to-first-token may not improve your product.

Prefill and Decode Behave Differently

LLM requests have two major compute phases.

Prefill

Processes the input prompt/context.

Long prompts make this phase expensive.

Decode

Generates output tokens iteratively.

This phase is often memory-bandwidth and KV-cache sensitive.

Modern serving systems increasingly optimize these phases separately. Current vLLM documentation includes techniques such as chunked prefill and disaggregated prefill/decode approaches.

This matters for workloads with extreme context lengths because a huge prefill can otherwise block smaller interactive requests.

Quantization Changes Capacity Economics

Serving models at reduced precision can shrink weight memory and sometimes improve throughput.

Modern runtimes support multiple quantization formats, including various FP8, INT8, INT4, GPTQ/AWQ-style and other hardware/model-specific options.

Quantization can allow:

larger model on same GPU
or
more replicas on same hardware
or
larger KV cache / concurrency

But it may also affect:

output quality
supported kernels
latency
hardware portability

Do not choose a format from a leaderboard. Validate the exact model, hardware, and task quality.

Multi-GPU Serving Is Not One Thing

When a model cannot fit efficiently on one accelerator, serving may use several parallelism strategies.

Tensor parallelism

Splits model operations across devices.

Pipeline parallelism

Splits layers/stages across devices.

Data parallelism

Replicates model execution so different requests can be handled by different replicas.

Expert parallelism

Relevant for mixture-of-experts models where expert computation is distributed.

Each changes communication cost and failure behavior.

A second GPU does not automatically give twice the throughput. Interconnect bandwidth and synchronization overhead matter.

Model Servers Need Backpressure

If traffic arrives faster than inference can run, requests accumulate.

A dangerous design is an effectively unbounded queue:

traffic spike
   ↓
queue grows
   ↓
latency explodes
   ↓
clients retry
   ↓
even more load

A production server should use:

bounded queues
request deadlines
admission control
concurrency limits
rate limits
overload responses

Sometimes rejecting a request quickly is more reliable than accepting it and responding two minutes later.

Autoscaling Should Follow Inference Pressure

CPU utilization is often a poor autoscaling signal for GPU inference.

A better signal may be:

pending requests
queue depth
concurrent sequences
KV-cache utilization
tokens waiting
GPU utilization
request latency

KServe’s current LLM autoscaling guidance supports KEDA-based scaling from Prometheus or OpenTelemetry metrics, including runtime signals such as waiting requests and KV-cache usage.

Scale on the bottleneck that actually predicts saturation.

Cold Starts Are Expensive for Large Models

Starting a model replica may require:

schedule container
→ load weights from storage
→ initialize runtime
→ allocate accelerator memory
→ build/initialize kernels/cache
→ become ready

For large models this can be much slower than a normal web-service startup.

Autoscaling policy must account for that delay.

Possible strategies include:

minimum warm replicas
predictive scaling
fast local/model caches
staged readiness
load-aware routing

Scale-to-zero is economically attractive but can be a poor fit for latency-sensitive large-model endpoints.

Rollouts Need Model-Aware Traffic Management

A new model version can change:

latency
memory consumption
output schema
quality
token usage
tool-calling behavior
safety characteristics

Do not replace the production version instantly.

Safer rollout patterns include:

shadow traffic
canary percentage
A/B test
progressive rollout
fast rollback

Current KServe releases include traffic-splitting and canary-oriented serving capabilities, which reflect how important model-aware rollout control has become.

Inference Graphs for Multi-Model Pipelines

Some predictions require more than one model.

Example:

request
  ↓
preprocessor
  ↓
classifier
  ↓
conditional specialist model
  ↓
postprocessor

KServe supports inference graphs for distributed model pipelines. Triton supports ensembles that connect preprocessing, model inference, and postprocessing while reducing unnecessary intermediate transfers.

Use a graph only when the workflow genuinely has multiple inference stages. Do not split one simple model endpoint into a distributed DAG for architectural aesthetics.

Observability Must Separate Waiting From Computing

“Request took 900 ms” is not enough information.

Break latency into stages:

network/gateway
queue
batch formation
prefill
model execution/decode
postprocessing
response serialization

Triton exposes metrics for request counts, queue time, inference execution, GPU utilization, CPU utilization, memory, and related serving behavior.

For LLMs also monitor:

time to first token
inter-token latency
output tokens/sec
prompt tokens/sec
active sequences
pending sequences
KV-cache occupancy
prefix-cache hit rate where relevant

This lets you distinguish “GPU is slow” from “requests are sitting in a scheduler queue.”

Capacity Planning Should Use Real Traffic Distributions

Average request length is dangerous.

An LLM endpoint may have:

median prompt: 1,500 tokens
p95 prompt: 40,000 tokens

The p95 workload can dominate memory and latency even if the average looks harmless.

Benchmark distributions for:

prompt length
output length
concurrency
tool-call loops
streaming duration
multimodal input size

Load tests should reproduce real traffic shape, not only send identical short prompts at maximum rate.

API Compatibility Helps, but Semantics Still Differ

Many modern LLM runtimes expose OpenAI-compatible APIs.

That is useful because clients can reuse request/response integrations.

But “compatible endpoint” does not guarantee identical behavior across runtimes for:

tool calling
structured output
reasoning fields
multimodal input
log probabilities
stop behavior
usage accounting
streaming edge cases

Test the capabilities your application depends on before assuming provider/runtime portability.

Security for Model Serving

Inference endpoints can be extremely expensive and may process sensitive data.

Protect them with:

authentication
authorization
tenant-aware quotas
request-size limits
rate limits
TLS
private network boundaries where appropriate
secret management
model/artifact integrity controls

Also treat model files as supply-chain artifacts. Verify where they came from and who can update the model repository.

When to Use Triton, vLLM, or KServe

These tools solve different layers.

vLLM

Strong fit when the main workload is generative LLM serving and you need capabilities such as continuous batching, KV-cache optimization, prefix caching, quantization, distributed inference, streaming, and OpenAI-compatible APIs.

NVIDIA Triton Inference Server

Strong fit for heterogeneous ML inference across multiple model frameworks, particularly when dynamic batching, multiple schedulers, ensembles, metrics, and GPU-oriented optimization matter.

KServe

Strong fit when Kubernetes-native lifecycle management, autoscaling, rollout control, inference graphs, and standardized serving resources are required around one or more runtimes.

They can also be combined. KServe can manage services whose underlying runtime is optimized for the specific model family.

Production Checklist

Final Takeaway

Model serving is where model quality meets systems engineering.

The production bottleneck is often not the neural network itself. It is the interaction between batching, queues, KV cache, accelerator memory, request length, autoscaling, model loading, routing, and rollout strategy.

Start with a clear latency budget and representative workload. Choose the runtime that matches the model family. Add orchestration only when you need lifecycle and scaling control. Then optimize with measured queue, GPU, and token-level data rather than theoretical hardware capacity.

Official References

Tags:Model ServingLLM InferencevLLMTritonKServeGPUMLOps

AI & AGENT ENGINEERING

BUILDING AN AI PRODUCT?

Talk to Lofingo about custom autonomous AI agents, enterprise workflow automation, and scalable software systems.

EXPLORE AI SERVICES →