LLM GPU memory calculator
Pick a model, a precision and the context length you need. See how much GPU memory it takes to run, split into model weights and KV cache, and how many GPUs that means.
Llama 3.3 70B, 4-bit
- weights
- KV cache (context)
- runtime and activations
GPUs needed
| GPU | Memory | GPUs needed |
|---|---|---|
| RTX PRO 4000 Blackwell | 24 GB | 3 |
| L40S / RTX 6000 Ada | 48 GB | 2 |
| H100 / A100 | 80 GB | 1 |
| RTX PRO 6000 Blackwell | 96 GB | 1 |
Estimate for inference (running the model). Fine-tuning needs much more memory. Real use varies by a few GB between runtimes such as vLLM, llama.cpp and Ollama.
GPU servers
A 70B model at 4-bit runs on one RTX PRO 6000 server with 96 GB of GPU memory. Smaller models fit the RTX PRO 4000 with 24 GB; the largest open models need the 4 × RTX PRO 6000 server with 384 GB. See all GPU servers.
How the memory is worked out
- Model weights = parameters × bits per weight ÷ 8. A 70B model takes about 141 GB at 16-bit, 75 GB at 8-bit and 43 GB at 4-bit (Q4_K_M uses about 4.85 bits per weight once its scales are counted). Mixture-of-experts models such as Qwen3 30B-A3B or gpt-oss-120b only use a few experts per token, but every expert must be loaded, so the total parameter count decides the memory.
- KV cache = 2 × layers × KV heads × head size × context length × users × bytes per value. It grows with every token of context and every user served at once, and for long contexts it can be larger than the model itself. Models with sliding-window layers (Gemma, gpt-oss) or compressed attention (DeepSeek's MLA) need far less.
- Runtime overhead: about 1.5 GB per GPU for the CUDA runtime, plus activations and buffers (the larger of 1 GB or 5% of the weights). A model counts as fitting when everything stays within 95% of the GPU memory.
Model figures come from each model's published configuration (checked 4 November 2026). The result is an estimate: vLLM, llama.cpp, Ollama and TensorRT-LLM differ by a few GB, and vLLM by default reserves 90% of GPU memory and fills the rest of it with KV cache. Fine-tuning needs several times more memory than running a model.
Which GPU server?
- RTX PRO 4000 Blackwell, 24 GB: models up to about 14B at 8-bit or 32B at 4-bit with modest context, embeddings, Whisper and image models.
- RTX PRO 6000 Blackwell, 96 GB: 70B models at 4-bit with long context, gpt-oss-120b, Qwen3 30B-A3B at 16-bit, or several smaller models at once.
- 4 × RTX PRO 6000, 384 GB: Llama 3.3 70B at 16-bit, Qwen3 235B at 8-bit, or high-throughput serving for many users.
Questions
How much VRAM do I need for a 70B model?
About 43 GB for the weights at 4-bit (Q4_K_M), plus the KV cache: roughly 2.7 GB for 8,192 tokens of context for Llama 3.3 70B. With runtime overhead, about 50 GB, so one 96 GB GPU is enough, with room for longer context.
Can a 24 GB GPU run a 32B model?
At 4-bit the weights take about 20 GB, which leaves little room for context. It works for short prompts; for long contexts or several users, a 48 GB or 96 GB GPU is more comfortable.
Why does context length use so much memory?
Every token in the context keeps its keys and values for every layer. For Llama 3.1 8B that is 128 KB per token at 16-bit, so a 128,000-token context needs about 17 GB on top of the model.
Does quantisation make the model worse?
Slightly. 8-bit is nearly identical to 16-bit; good 4-bit formats such as Q4_K_M or AWQ usually lose a little accuracy and are the most popular choice for self-hosting. Below 4 bits quality drops faster.
Can I split a model over several GPUs?
Yes. vLLM, TensorRT-LLM and llama.cpp can split a model across GPUs in one server (tensor or pipeline parallelism). Each GPU then needs its own runtime overhead, which the calculator includes.
Not sure which server fits?
Tell us what you want to run. We reply around the clock, usually within the hour.