A team I advised rented a GPU instance to serve a model behind an internal tool. Reserved, always on, about $1,100 a month. When we looked at the metrics, the GPU was doing actual work for roughly forty minutes a day. They were paying for 24 hours of capacity to use half an hour of it, and the reason was simple: the alternative — cold starts of two to three minutes while a model loaded — made the tool unusable.
That trade-off, pay for idle or wait for cold start, is the entire economics of AI inference infrastructure. Serverless GPUs exist to break it, and they mostly do, provided you understand what actually causes the cold start.
Why GPU Cold Starts Are Different
A serverless CPU function cold-starts in a couple of hundred milliseconds. A GPU container doing the same thing can take minutes, and it is worth knowing where that time goes because every optimisation targets one of these:
Getting a machine. GPU capacity is scarcer than CPU. Sometimes you wait for the provider to find one.
Pulling the image. A container with CUDA, PyTorch and dependencies is commonly 5-15GB. That download dominates on a cold node.
Loading the weights. A 7B model in fp16 is around 14GB moving from storage into GPU memory.
Warming up. Compiling kernels, allocating buffers, running a first pass that is slower than every subsequent one.
Notice that only the last one is about your code. The first three are about size, which is why the fixes are mostly logistical rather than clever.
The Things That Actually Reduce It
Separate the weights from the image. The most common mistake is baking model weights into the container. Now every cold start downloads a 20GB image. Keep the image lean and pull weights from a fast network volume or a cache the platform provides — most serverless GPU platforms have a mechanism for exactly this, and it is the single biggest win available.
Load the model once, outside the handler. Obvious, still missed. Module-level initialisation runs once per container; handler-level runs per request.
import modal
app = modal.App("classify")
image = modal.Image.debian_slim().pip_install("torch", "transformers")
volume = modal.Volume.from_name("model-cache") # weights live here, not in the image
@app.cls(gpu="A10G", image=image, volumes={"/cache": volume},
scaledown_window=300, min_containers=0)
class Classifier:
@modal.enter() # runs once per container, not per request
def load(self):
from transformers import pipeline
self.pipe = pipeline("text-classification", model="/cache/model")
@modal.method()
def run(self, texts: list[str]):
return self.pipe(texts, batch_size=32) # batch — the GPU is parallel
Keep containers alive longer than you think. That scaledown_window is the most important number in the config. Traffic that arrives in clusters — which is most internal-tool traffic — means a 5-minute idle window turns dozens of cold starts into one. Idle GPU time is cheaper than an unusable tool.
Batch requests. A GPU processing one item at a time is mostly idle silicon. Collecting requests over a short window and running them together often gives near-linear throughput gains for a small latency cost.
Quantise. An 8-bit or 4-bit model is a smaller download, loads faster, and fits on cheaper hardware. Evaluate the quantised build against your eval set rather than assuming quality is unchanged, but the trade is usually favourable.
Choosing Where It Runs
Four shapes, and the right one is usually decided by traffic pattern rather than preference.
A hosted model API. Still the correct default. No infrastructure, no idle cost, someone else handles capacity. Use this unless you have a specific reason not to.
Serverless GPU platforms — Modal, Replicate, RunPod, Baseten and similar. The right answer for a custom or fine-tuned model with spiky traffic. You pay per second of execution and scale to zero. This is what that team should have started with.
Managed inference on a cloud provider — SageMaker, Vertex, Azure ML. Heavier, more configuration, and worth it when you are already deep in one cloud and need it inside your VPC for compliance reasons.
Your own GPUs. Only sensible with sustained, predictable, high utilisation. The crossover is real — above roughly 50-60% utilisation, reserved capacity beats per-second billing — but you are now responsible for drivers, capacity planning and someone being on call for hardware.
Match the Card to the Model
People reach for the biggest GPU available and then wonder about the bill. The rough rule for memory: parameters × 2 bytes in fp16, plus room for the KV cache and activations. A 7B model needs roughly 14GB before overhead, so it fits comfortably on a 24GB card and is wasted on an 80GB one.
Cheaper cards — the mid-tier inference-oriented ones — are frequently the correct choice for models under about 13B, and cost a fraction of the flagship data-centre parts. The flagship cards are for training and for very large models, and most teams serving a fine-tuned 7B do not need them.
When You Do Not Need a GPU at All
Worth checking before any of the above. Embedding models, small classifiers and most models under about 1B parameters run acceptably on CPU, especially with a runtime like ONNX Runtime and a quantised model.
I have moved two embedding workloads from GPU to CPU and cut cost substantially with latency that was still well inside the requirement. Nobody had measured; the GPU was assumed. Measure first — this is the cheapest optimisation in the entire category.
What I Would Do Now
Start with a hosted API. If you genuinely need your own model, go serverless GPU with weights on a cached volume and a generous scale-down window. Batch wherever the workload allows. Only look at dedicated capacity when your utilisation graph justifies it, and check whether the thing even needs a GPU before renting one.
That team moved to a serverless platform with a 5-minute idle window. Their bill went from about $1,100 a month to under $60, and the tool got faster because the new setup had cached weights and their old always-on instance had been quietly swapping. The expensive option had not even been buying them the thing they were paying for.
