LLM VRAM Calculator
186 LLM architectures · 168 GPUs · inference & fine-tuning
Lower precision = less VRAM, usually faster inference
Max for this model: 128K
Global avg ≈ 0.47 · US ≈ 0.38 · EU ≈ 0.25 · France (nuclear) ≈ 0.06
Memory breakdown
Tap a row to see where the number comes from.
What drives this number
- The model does not fit, so the throughput figures below describe what this hardware would do if it did — add GPUs, quantize further, or shorten the context first.
Carbon footprint
How this is calculated
Every figure comes from the model’s published architecture — layer count, hidden size, FFN width, KV head geometry, vocabulary — rather than from parameter-count rules of thumb. Memory numbers are arithmetic; throughput and power are estimates and are labelled as such.
Weights
Quantization only applies to the transformer blocks. Embedding and output-projection matrices stay at FP16 in every serving stack, so they are priced separately — which is why a 4-bit model with a 256K vocabulary never gets as small as “params ÷ 4”. Multimodal checkpoints carry a vision or audio encoder on top (+15%, +40% for video).
(params − embeddings) × bytes/weight + embeddings × 2 BKV cache
Sized from the real attention geometry, not from the parameter count. GQA caches num_kv_heads × head_dim per layer instead of the full hidden size; MLA caches one compressed latent per layer; sliding-window layers stop growing past the window; linear-attention (Gated DeltaNet, Mamba) layers keep no cache at all. This is the term that decides whether long context fits.
2 × layers × kv_heads × head_dim × bytes × context × concurrent usersActivations
Serving engines prefill in 2048-token chunks, so only one chunk of hidden states is live at a time — inference activations barely grow with context. Training is the opposite: every layer’s intermediates stay resident until the backward pass, unless gradient checkpointing recomputes them.
inference: 6 × min(context, 2048) × hidden × bytes × batchOptimizer & gradients
Full fine-tuning pays 12 bytes per parameter with mixed-precision AdamW (8 B of moments + a 4 B FP32 master copy), plus one gradient per parameter. 8-bit Adam, Adafactor, SGD, pure BF16 and paged optimizers each change that figure. LoRA trains rank × hidden parameters per layer instead, which is why it collapses the cost.
full: (states + master) × params + gradients + 0.3 × weightsThroughput
Decoding one token streams every resident weight plus the KV cache through memory exactly once, so tokens/second tracks memory bandwidth — not FLOPS. Real stacks sustain about 80% of peak bandwidth, with a ~1.5 ms per-step floor. MoE models only read the experts they route to, so they decode like a much smaller model while still occupying memory like a large one.
ms/token ≈ (weights + KV) ÷ (bandwidth × 0.8) + 1.5 msTime to first token
Prefill is the compute-bound half: two FLOPs per active parameter per prompt token, plus the quadratic attention term that dominates long prompts. It shards across GPUs far better than decoding does.
TTFT ≈ 2 × active params × prompt tokens ÷ (TFLOPS × 0.75) + attentionSizing is a planning estimate. Reserve headroom for fragmentation, CUDA graphs and the serving engine’s own allocator — vLLM and SGLang pre-allocate a fixed share of VRAM for the KV pool.
Chat with Llama 3.1 8B — no setup
Skip the numbers and just try it. Our chat opens Llama 3.1 8B (and any close matches) so you can send a prompt and see how it responds right away.
Popular combinations
Pre-computed VRAM, throughput and hourly price for the most searched LLM × GPU pairs.