Llama 3.1 8B
8B
~56 tok/s per stream
DetailsHide details
Where the memory goes
- Model weights4.39 GB
- KV cache4.29 GB
- Activations0.03 GB
- CUDA & framework1.04 GB
- Precision
- INT4 / AWQ
- Serving preset
- 8K context × 4 users
- Users at this context
- 5
- Spare context in leftover VRAM
- 34K
- Fit score
- 56/100 · High