Llama 3.3 70B VRAM Calculator

TL;DR

Llama 3.3 70B requires 95.5 GB at fp8 and 56.3 GB at int4. A single RTX 4090 (24 GB) cannot run it at any quantization level. The RTX 5090 (32 GB) reaches the int4 minimum only as a dual-card NVLink pair (64 GB combined), hitting 47.7 tokens/sec at 87.9% use. Source: GPUniq benchmark data, 2026-09-21.

Does Llama 3.3 70B fit an RTX 4090?

No. Not even close.

The RTX 4090 has 24 GB of VRAM. Int4 - the most compressed practical option - needs 56.3 GB. That's more than double a single card. Fp8 needs 95.5 GB, a 4x gap. The GPUniq data does show an RTX 4090 running Llama 3.3 70B at fp8 with 24.5 tokens/sec, but that's a multi-GPU setup where four 4090s collectively provide 96 GB. One card alone is simply out.

No quantization level gets the model below 24 GB.

The 4090 is genuinely excellent for smaller models. Llama 3.1 8B at fp16 fits in 21.4 GB with 89.3% use and 41.3 tokens/sec on a single card. For 70B you're in the wrong weight class, and buying four 4090s to hit 96 GB costs more than a single A100 80GB.

VRAM by quantization level

The RTX 5090 has 32 GB physically. The GPUniq int4 run showing 64 GB available is a dual-5090 NVLink configuration. A single 5090 still can't hold int4 alone.

For fp16 (~140 GB) you need dual A100 80GB or an H100 NVL. Int8 (~70 GB) fits a single A100 80GB with headroom. Int4 at 56.3 GB fits a single A100 80GB at 62.9% use - that's where single-card deployment becomes possible.

RTX 4090 vs RTX 5090 throughput

From GPUniq benchmark data (2026-09-21):

• RTX 4090 at fp8: 24.5 tokens/sec, 99.5% use
• RTX 5090 at int4: 47.7 tokens/sec, 87.9% use

95% more throughput on the 5090 setup. The use gap is the real story. At 99.5%, the 4090 configuration has zero headroom for batching, KV cache growth, or concurrent requests. One long context window and you're swapping. The 5090 at 87.9% is tight but workable for single-user inference.

Against an H100: 47.7 vs 54.5 tokens/sec, a 12.6% gap. The H100 also has 80 GB available versus 64 GB, running at 62.9% use with room to batch several requests simultaneously. That compounding effect matters more than per-token speed at production scale.

Speed-to-memory tradeoff

Int4 wins for the 70B weight class. Fp8 at 95.5 GB gives 24.5 tokens/sec on a memory-saturated 4090 setup. Int4 at 56.3 GB gives 47.7 tokens/sec on the 5090 setup. That's 41% less memory and roughly double the throughput. The quality cost is real but well-studied; for most chat and instruction-following tasks, int4 is acceptable.

Fp16 at ~140 GB is for research workflows requiring exact numerics. You're paying 2.5x the memory of int4 for marginal quality gains in most benchmarks. Unless you're fine-tuning or doing activation analysis, fp16 on a 70B model is hard to justify operationally.

The H100 at int4 hits 54.5 tokens/sec with 45% memory headroom. That means batching 4-6 concurrent requests while staying under the ceiling, which changes the economics entirely. A single H100 serving batched int4 inference will outperform two RTX 5090s serving sequential requests.

Qwen2.5-72B vs Llama 3.3 70B

Qwen2.5-72B is slightly heavier across every configuration.

Qwen2.5-72B int4 needs 62.9 GB versus Llama 3.3 70B's 56.3 GB, a 6.6 GB difference. On the 5090 setup that pushes use to 89.7% vs 87.9% - close enough that it doesn't change your hardware decision. On H100 the difference is negligible.

The interesting row is the RTX 4090 one. Qwen2.5-72B runs int4 there at 65.6% use and 38.9 tokens/sec, better than Llama 3.3 70B's 24.5 on the same platform. Lower use means more headroom, which translates directly to throughput. Alibaba's Qwen2.5 model card notes denser attention patterns, which may explain the slightly higher memory footprint despite the speed advantage.

Minimum GPU for 70B inference

Single consumer card: nothing. No single consumer card holds 56.3 GB.

Multi-card consumer minimum: two RTX 5090s (64 GB combined) at int4, 47.7 tokens/sec. Use is tight and batching headroom is minimal.

Single-card enterprise minimum: the A100 40GB cannot hold 56.3 GB. You need the A100 80GB, which runs int4 at 50.3 GB required, 62.9% use, approximately 35.4 tokens/sec. That's slower than the dual-5090 setup because the A100's memory bandwidth is lower than the H100's despite equal capacity.

Production standard: H100 80GB at int4, 54.5 tokens/sec, 62.9% use, enough headroom to batch requests and handle long contexts without swapping.

RTX 4090 with CPU offloading via llama.cpp's --n-gpu-layers partial offload is technically possible. Expect 2-5 tokens/sec with heavy PCIe thrashing. That's a demo, not a deployment.

GPU comparison matrix

GPUniq benchmark data (2026-09-21), Llama 3.3 70B inference, single-stream generation:

The L40S and RTX 6000 Ada numbers surprised me. Both have 48 GB, which sounds like useful headroom, but they're stuck at fp8 across multiple cards and produce only 15.1 and 16.7 tokens/sec - worse than the four-card 4090 setup. Memory bandwidth is the constraint: neither card is optimized for the high-bandwidth access pattern a 70B model demands. The H100's HBM3 memory is why it pulls 54.5 tokens/sec on the same int4 weights that give the A100 only 35.4.

If you're choosing between an L40S and an A100 80GB for 70B inference, take the A100 despite it being older hardware.

How to use this calculator

Work backwards from total available VRAM. Under 56.3 GB combined, int4 is out. Under 95.5 GB, fp8 is also out on that configuration.

Required VRAM (GB) = model_params * bytes_per_param * overhead_factor

fp16:  70B * 2 bytes  * 1.2 = ~168 GB  (rough; measured ~140 GB)
int8:  70B * 1 byte   * 1.2 = ~84 GB   (measured ~70 GB)
fp8:   70B * 1 byte   * 1.1 = ~77 GB   (measured 95.5 GB with KV cache)
int4:  70B * 0.5 byte * 1.2 = ~42 GB   (measured 56.3 GB with KV cache)

Measured values exceed the naive formula because KV cache, activations, and framework overhead add 10-30% on top of weight storage. GPUniq figures include a realistic KV cache for a 2048-token context window.

To use the interactive calculator:

1. Select your GPU from the dropdown: RTX 4090, RTX 5090, H100, A100, L40S, or RTX 6000 Ada.
2. Select the model: Llama 3.3 70B, Qwen2.5-72B, Llama 3.1 8B, or DeepSeek-V3 671B.
3. Choose quantization. The calculator flags configurations where required GB exceeds available GB.
4. Check use. Above 90% means no practical headroom for batching. Below 70% is comfortable for production.

Tokens/sec figures are from single-stream generation with a 512-token prompt and 256-token output. Batch inference will show higher throughput per token but higher latency per request.