Running LLM Inference on Kubernetes: A Small Team's Guide

By 2026, Kubernetes has become the default orchestration layer for AI infrastructure across most engineering organizations — GPU scheduling, distributed training coordination, and LLM inference serving are now supported by a mature ecosystem of operators and schedulers, not the bespoke, hand-rolled setups they required a few years ago. That maturity is genuinely good news for a small team: self-hosting LLM inference is more approachable than it used to be, but “more approachable” still means there are specific decisions worth getting right before you deploy, because the failure modes here — cost overruns and latency problems — are both expensive and slow to notice if you’re not watching for them.

The first decision: should you be self-hosting at all?

This is worth answering honestly before anything else. Self-hosting makes sense when you need data to stay within your own infrastructure for compliance reasons, when you’re running high enough volume that API costs from a hosted provider exceed the infrastructure and operational cost of self-hosting, or when you need a model or fine-tune that isn’t available through a hosted API. If none of those apply, a hosted inference API is very likely cheaper and less operationally risky for a small team than standing up and maintaining your own serving infrastructure — self-hosting is a real commitment, not a default best practice.

GPU scheduling: the part that’s genuinely different from typical workloads

Inference serving has a different resource profile from typical web services, and treating it identically tends to produce either wasted spend or poor latency.

Model size determines your GPU memory floor, non-negotiably. Unlike CPU/memory requests for a typical service, there’s no meaningful way to under-provision GPU memory for a model that needs it — it either fits or it doesn’t. Get this number right from the model’s actual requirements, not a guess, before worrying about anything else.

Fractional GPU scheduling matters more here than almost anywhere else. If you’re serving multiple smaller models or handling modest request volume, a single inference workload rarely needs a full high-end GPU continuously. Dynamic Resource Allocation, now generally available, is what makes efficiently sharing a device across inference workloads a native scheduling option rather than a manual workaround.

Cold start time is a real latency concern, distinct from typical pod startup. Loading a large model into GPU memory takes meaningfully longer than a typical container starting up — this affects both your autoscaling configuration (a scale-up event needs more lead time than a typical web service) and your choice of minReplicas, which often needs to be higher than 0 or 1 specifically to avoid cold-start latency hitting real user requests.

Autoscaling inference workloads specifically

Standard Horizontal Pod Autoscaling on CPU utilization is usually the wrong metric for inference serving — GPU workloads’ bottleneck is GPU utilization or request queue depth, not CPU. Scaling on the wrong metric here produces the same “HPA looks broken” experience as it does elsewhere, just with a more expensive resource sitting idle or contended as a result. Custom metrics based on request queue depth or GPU utilization directly are worth the initial setup effort for inference workloads specifically, more so than for typical CPU-bound services.

Cost control specific to LLM inference

Token-based or request-based cost attribution, not just infrastructure cost. Knowing your GPU spend is one thing; knowing the cost per request or per token served is what actually lets you evaluate whether self-hosting is beating a hosted API’s per-token pricing, which is usually the comparison that matters for the ongoing “should we still be self-hosting” question.

Batch requests where latency requirements allow it. Inference throughput improves substantially with batching for many model architectures — if your use case can tolerate slightly higher per-request latency in exchange for meaningfully better throughput per GPU, this is one of the highest-leverage cost optimizations available and is often left on the table simply because it requires deliberate request-handling design rather than a scheduling change.

Right-size the model to the task, not just the GPU to the model. A smaller, fine-tuned model that’s good enough for your specific task is often both cheaper to serve and lower-latency than routing everything through your largest available model by default — this is a modeling decision more than an infrastructure one, but it has larger cost implications than most infrastructure tuning.

Monitoring: what actually matters for inference specifically

Beyond standard infrastructure health, inference serving benefits from watching latency percentiles specifically (p95/p99 matter enormously here, since tail latency in LLM serving can be dramatically worse than median due to variable-length generation), GPU memory headroom (an out-of-memory error mid-generation is a worse failure mode than a typical OOM, since it can lose a partially-generated response), and queue depth as a leading indicator of when to scale before latency actually degrades.

This is where general Kubernetes fleet monitoring and inference-specific concerns overlap directly: VantraOps’ metrics and health checks cover the infrastructure-layer signals — GPU and memory utilization, restart patterns, node health — that inference workloads need watched just as much as any other production service, without requiring a separate, specialized ML observability stack layered on top for the infrastructure half of the picture.

Frequently asked questions

Do I need Kubernetes-specific ML tooling (KServe, Ray Serve) or is plain Kubernetes enough? For simple single-model serving, a standard Deployment with an appropriately configured resource spec and autoscaler is often sufficient. Specialized serving frameworks earn their added complexity once you’re managing multiple models, need advanced traffic-splitting for A/B testing, or need features like request batching handled for you rather than built manually.

How much GPU memory headroom should I leave above a model’s stated requirements? Enough to cover the KV cache growth during generation, which scales with context length and batch size — this is often underestimated because the model’s base memory footprint is well-documented but the runtime overhead during actual generation is workload-specific and needs to be measured, not assumed.

Is self-hosting ever cheaper than hosted APIs at small scale? Rarely at genuinely small volume, once infrastructure and operational cost are counted honestly — the crossover point where self-hosting wins tends to require sustained, meaningful request volume, not sporadic or low-traffic usage.

Does autoscaling to zero make sense for inference workloads? Only if cold-start latency is acceptable for your use case — for anything user-facing and latency-sensitive, a minReplicas of at least 1 (accepting some idle cost) is usually the better tradeoff than scaling to zero and making users wait through a model load on the first request after an idle period.

See infrastructure health across your GPU and general compute fleet in one place — start free with one cluster.