Your ML team requested four A100 GPUs for training. They got them. Utilisation is 12%. The GPUs sit idle 20 hours a day while your cloud bill climbs five figures a month.
This isn't unusual. Most ML teams overspend on infrastructure because they optimise for capability, not efficiency. They provision for peak training runs and pay for idle time. They use the largest instance available because "it's faster." They never measure cost per prediction or cost per experiment.
ML infrastructure cost optimisation isn't about cutting corners. It's about eliminating waste so you can invest more in the work that matters.
Before optimising anything, know where your spend lands. Most teams are surprised.
| Cost category | Typical share | Often overlooked |
|---|---|---|
| Training compute (GPU/CPU) | 30-50% | Idle time between runs, oversized instances |
| Inference/serving | 20-40% | Always-on endpoints with low traffic, no autoscaling |
| Storage | 10-20% | Duplicate datasets, uncompressed checkpoints, abandoned experiments |
| Data processing | 5-15% | Redundant feature computation, unoptimised Spark jobs |
| Networking | 2-5% | Cross-region data transfer, large model artifact pulls |
Run a cost audit by tagging resources to teams and projects. Cloud providers offer cost allocation tags. Use them. If you can't attribute a cost to a team, nobody owns it, and nobody will reduce it.
If you can't attribute infrastructure cost to a team or project, nobody owns it and nobody will reduce it. Tag everything.
GPU utilisation below 30% is money on fire. Common fixes:
Right-size instances. Not every training job needs an A100. Profile your workload. Many jobs are memory-bound, not compute-bound, and run fine on smaller GPUs at a fraction of the cost. Use T4s or L4s for experimentation. Reserve large GPUs for final training runs.
Spot instances. Training workloads are interruptible. Spot instances cost 60-90% less than on-demand. Implement checkpointing so interrupted jobs resume rather than restart. AWS Spot, GCP Preemptible VMs, and Azure Spot VMs all support this pattern.
Autoscaling training clusters. Scale to zero when no jobs are running. Kubernetes with node autoscaling (Karpenter on AWS, GKE Autopilot on GCP) handles this. Don't pay for a GPU cluster that runs one job per day.
Hyperparameter search budgets. Unbounded grid searches are the fastest way to burn GPU hours. Use Bayesian optimisation (Optuna, Ray Tune) with early stopping. Set hard budget limits: maximum number of trials, maximum total GPU hours.
Mixed precision training. FP16 or BF16 training uses half the memory and runs 2-3x faster on modern GPUs with negligible accuracy impact. If your training script isn't using mixed precision, you're overpaying by default.
Spot instances with checkpointing can reduce training costs by 60-90%. If your training jobs aren't on spot, start there.
Training costs are visible (big bills during training runs). Inference costs are insidious (small per-request costs that compound with traffic).
Autoscale to zero. Low-traffic endpoints don't need always-on compute. Scale inference endpoints to zero replicas during off-hours and spin up on first request. Cold start latency is acceptable for many batch and internal use cases.
Right-size serving instances. An inference endpoint serving a 200MB scikit-learn model doesn't need a GPU. CPU inference is 10-50x cheaper per instance. Only use GPU serving for models that genuinely need it (large neural networks, real-time latency requirements).
Batch predictions where possible. If your use case tolerates hourly or daily predictions instead of real-time, batch inference is dramatically cheaper. Run predictions overnight on spot instances, store results, serve from cache.
Model optimisation. Quantisation (FP32 to INT8) reduces model size 4x and inference latency 2-3x. Distillation replaces a large model with a smaller one that performs nearly as well. Both reduce per-request cost directly.
| Optimisation | Compute reduction | Accuracy impact | Effort |
|---|---|---|---|
| Quantisation (INT8) | 2-4x | Less than 1% for most models | Low |
| Distillation | 5-20x | 1-3% depending on task | High |
| Pruning | 2-5x | Variable | Medium |
| Compilation (TensorRT, ONNX) | 1.5-3x | None | Low |
ML teams accumulate storage without cleaning up:
Traditional FinOps practices need adaptation for ML workloads:
Unit cost tracking. Cost per experiment, cost per training run, cost per 1000 predictions. These metrics make waste visible to the team, not just to finance.
Budget alerts per team. Set weekly or monthly spend thresholds with automated alerts. Catch runaway jobs (the accidental 1000-trial hyperparameter search) before the invoice arrives.
Chargeback or showback. Allocate costs to the teams that incur them. Even showback (visibility without billing) changes behaviour. Teams that see their spend reduce it.
Regular cost reviews. Monthly reviews of the top 10 cost line items. Ask: is this still needed? Is this right-sized? Could this be spot? Make cost a first-class engineering concern, not a finance concern.
Treat cost per prediction and cost per experiment as engineering metrics, not finance metrics. Make them visible on the team dashboard alongside latency and accuracy.
If you do nothing else, do these five things:
These five changes typically reduce ML infrastructure costs by 30-50% within the first month.