GPU Rightsizing: The Biggest Kubernetes Cost Lever Nobody's Using
CPU and memory rightsizing gets most of the attention in Kubernetes cost optimization content, for a reasonable reason — it applies to nearly every cluster. GPU rightsizing gets far less coverage despite being, by most current estimates, the single largest underexploited cost lever in Kubernetes clusters that run any AI or ML workload. The reason is straightforward: GPUs are expensive enough per unit that even teams with no formal cost discipline elsewhere still notice the bill, but scarce enough — and scheduling for them historically primitive enough — that most teams have had no practical way to see or fix the waste even once they’ve noticed it.
Why GPU waste is bigger, proportionally, than CPU waste
A moderately over-provisioned CPU request costs you some fraction of a modestly-priced VM. A moderately over-provisioned GPU request costs you some fraction of the most expensive line item in your infrastructure bill, often by a wide margin. The same percentage of waste translates to a dramatically larger dollar figure — which is exactly why this specific lever deserves attention disproportionate to how often it comes up in general Kubernetes cost content.
Compounding this, GPU scheduling has historically been more primitive than CPU/memory scheduling: the legacy Device Plugin model could only express GPU requests as whole-integer counts, with no native way to share a device across multiple lighter workloads. A team running several small inference services had no built-in mechanism to pack them efficiently — each claimed a whole GPU, whether it needed the whole thing or not. Dynamic Resource Allocation, which reached general availability in Kubernetes 1.34, is what’s finally making fractional sharing and smarter bin-packing a native, practical option rather than a workaround.
Finding the waste: what to actually check
Utilization against allocation, per GPU, over a real time window. A GPU sitting at 20% utilization while fully allocated to one workload is the clearest, easiest-to-find waste — if you’re not already tracking this per-device, it’s the first gap to close, because everything else on this list depends on being able to see it.
Whether training and inference workloads are sharing pools efficiently, or unnecessarily siloed. Training jobs are often bursty — intense for hours or days, then idle — while inference serving tends to be steadier but lighter per-request. Pools that don’t allow these to share capacity, when they safely could, leave one or the other idle a meaningful fraction of the time.
Whether idle GPU capacity is being kept “warm” out of inertia rather than genuine need. Provisioning ahead of an anticipated training run is a legitimate reason to hold idle capacity temporarily — permanently holding capacity “in case” a run happens with no scheduled need is a different thing, and worth distinguishing honestly from the former.
Whether smaller models or lighter inference workloads actually need a full high-end GPU, or would run fine on a smaller instance type. It’s common for a workload to be provisioned on whatever GPU type is already standard in the fleet, rather than the one that actually matches its resource profile — a right-sizing check at the instance-type level, not just the utilization-within-a-device level.
What to actually do once you’ve found it
Adopt fractional GPU scheduling for workloads that don’t need a whole device. This is the highest-leverage fix now that DRA-based scheduling makes it a native option — see our DRA and GPU scheduling piece for the specifics of what changed.
Use priority and preemption to let training and inference share pools safely, rather than keeping them in permanently separate, individually over-provisioned capacity. The scheduler can make deliberate tradeoffs under contention if you tell it how to, rather than requiring pre-siloed capacity to avoid any contention at all.
Right-size instance type, not just utilization within a device. A workload consistently running well under a smaller GPU type’s capacity is a signal to downsize the instance type, not just tolerate the waste because the workload technically fits.
Re-check quarterly, same as every other rightsizing exercise. Model sizes change, traffic to inference endpoints shifts, and a GPU allocation that was correctly sized six months ago may not be today — this isn’t a one-time cleanup, it’s a recurring discipline, same as CPU and memory rightsizing.
Why this needs actual visibility, not intuition
GPU waste is particularly hard to catch by feel, because “the GPU bill is high” doesn’t tell you whether that’s because you’re running a lot of legitimately necessary work, or because a handful of workloads are sitting mostly idle on expensive, fully-allocated devices. Cost mapped to real utilization, broken down per workload rather than as an aggregate bill, is what turns that ambiguity into a specific, actionable finding — which workload, which device, how much headroom is actually sitting unused.
Frequently asked questions
Is GPU rightsizing only relevant for teams doing model training? No — inference serving is often the larger, more consistent source of GPU spend for teams that aren’t training their own models, and it benefits from the same rightsizing discipline, arguably more, since inference workloads run continuously rather than in bursts.
Does fractional GPU sharing risk performance problems for latency-sensitive inference? It’s a real consideration, not a non-issue — sharing a device across workloads means contention is possible under load. This is a case where visibility into actual utilization patterns matters before deciding which workloads are safe to share versus which need dedicated capacity.
How much waste is typical for teams that haven’t looked at this yet? It varies widely by workload mix, but given that GPU rightsizing is considered the largest underexploited lever in current Kubernetes cost data, teams that haven’t specifically audited this are more likely than not to find meaningful headroom once they look.
Is this worth addressing before or after adopting DRA-based scheduling? Worth starting the visibility work immediately regardless — knowing where the waste is doesn’t require DRA. Acting on fractional sharing specifically does benefit from DRA-based scheduling being in place, so the two efforts pair naturally rather than needing to happen in strict sequence.
See real GPU and compute utilization mapped to cost, not list-price guesses — start free with one cluster.