GPU VRAM Calculator for Inference LLM on-Premise
Estimate the required VRAM for running an LLM model locally. Choose the number of parameters, quantization (FP16, INT8, INT4), and get the recommended tier GPU for your hardware on-prem.
Configure
Result
No results
Enter values and press the button to calculate.
Frequently Asked Questions
How is the required VRAM calculated for inference of an LLM?
Approximate estimate used by the ML community: VRAM_GB = parameters × bytes_per_param × overhead_factor. For an FP16 model with 7 billion parameters: 7 × 2 × 1.2 = 16.8 GB. This 20% overhead covers KV cache, intermediate activations, and framework footprint (PyTorch, llama.cpp, Ollama). The estimate is approximate: architectures like MoE, Mamba or vision transformer may require additional VRAM.
What is the difference between FP16, INT8 and INT4 for inference?
FP16 (float at 16 bits) is the full precision standard: each parameter occupies 2 bytes. INT8 (8-bit quantization) halves VRAM consumption with minimal loss of quality (<1% in main benchmarks). INT4 (4-bit quantization, e.g., GGUF Q4_K_M) reduces VRAM to 25% of FP16 while maintaining acceptable quality for most tasks. The optimal choice depends on available hardware and task sensitivity to quality.
What GPU is required to run Llama 3 70B locally?
Llama 3 70B in FP16 requires approximately 168 GB of VRAM (beyond the reach of a single consumer GPU). In INT8, it drops to around 84 GB (1× NVIDIA H100 80 GB or 2× A100 40 GB). In INT4, about 42 GB are sufficient (e.g., 2× RTX 3090 24 GB in tensor parallel, or 1× A100 80 GB). For practical on-prem use with a budget-friendly setup, consider Llama 3.1 8B (INT4: ~4.8 GB) or Llama 3 13B (INT4: ~7.8 GB) on consumer GPUs.
What is the KV cache and how does it affect VRAM?
Key-Value Cache (KV Cache) stores pre-computed tokens during generation to avoid recalculation at every step. Its size depends on the context (number of tokens): for long contexts (32k-128k tokens), KV cache may require more VRAM than the model itself. The default overhead of 1.2 (20%) is conservative for medium contexts; consider an overhead of 1.5–2.0 for contexts > 32k tokens. Techniques like FlashAttention-2 and KV cache quantization (GQA, MQA) significantly reduce this impact.
Can I run a 13B LLM on an RTX 4090 with 24GB of VRAM?
Yes, with proper quantization. A model 13B in INT4 requires approximately 7.8 GB (13 × 0.5 × 1.2), widely compatible with 24 GB. In INT8 it goes up to ~15.6 GB, still within the limits of the RTX 4090. In FP16 it would require ~31.2 GB, exceeding the available 24 GB. Tools like Ollama, llama.cpp and GPTQ make automatic quantization transparent. The RTX 4090 is one of the most efficient consumer GPUs for local inference: high VRAM (24 GB) + high memory bandwidth (1 TB/s) makes it ideal for models 7B–30B quantized.
How is it used?
- Insert number of parameters
Indicate the number of model parameters in billions (B). Examples: Llama-3-8B = 8, Mistral-7B = 7, Qwen2.5-72B = 72.
- Select Quantization
FP16 offers the highest quality but requires double the VRAM compared to INT8. INT4 is most memory-efficient but slightly reduces output quality.
- Set overhead factor
Default value 1.2 represents a 20% overhead for KV cache, activations and frameworks. Increases to 1.3-1.5 for high batches or long contexts.
- Calculate and interpret the result
Calculator shows estimated VRAM in GB and recommended tier GPU (consumer, prosumer, professional, datacenter). Use the result to choose the appropriate on-prem hardware.
RAM and inference LLM on-prem: how to estimate required hardware
Run a Large Language Model (LLM) locally - on an on-prem server, dedicated VPS, or GPU-enabled workstation - requires mainly sufficient VRAM (Video RAM).
The standard empirical formula for community ML is: VRAM_GB = parametri_B × bytes_per_param × overhead_factor. For a 7 billion parameter model (7B) in FP16, the calculation is: 7 × 2 × 1.2 = 16.8 GB. The 20% overhead covers KV cache (token memory already processed), intermediate activations and framework footprint (Ollama, llama.cpp, vLLM).
Quantization is the most important factor to reduce VRAM requirements without sacrificing quality entirely: INT8 halves VRAM compared to FP16 with minimal loss of quality; INT4 reduces to 1/4 of FP16 with still acceptable quality for most production tasks. Tools like GGUF/llama.cpp, GPTQ and AWQ make quantization automatic and accessible to anyone.
Warning: This estimate is empirical (medium confidence, A4) and may under-estimate the real requirement for special architectures like Mixture-of-Experts (MoE, e.g., Mixtral), Vision models (e.g., LLaVA), or models with very long contexts (>32k tokens). In these cases, increase the overhead factor to 1.5–2.0.
Practical example: Llama 3.1 8B in INT4 on RTX 4090
- Model: Llama 3.1 8B (8 billion parameters)
- Quantization: INT4 (0.5 bytes per parameter) - format GGUF Q4_K_M
- Overhead factor: 1.2 (20% per KV cache + activations)
- RAM = 8 x 0.5 x 1.2 = 4.8 GB
- GPU Tier: Consumer (≤8GB) - RTX 3080/4080 with wide margin
Dictionary - VRAM and LLM Inference
- Video RAM
- Video RAM - dedicated memory of the GPU. Model parameters for LLM are loaded into VRAM for inference: without enough VRAM, the model won't start or uses system RAM (much slower).
- Quantization
- Technique to reduce numerical precision of parameters from 32/16 bit to 8 or 4 bits, drastically reducing VRAM needed with minimal impact on output quality. Common formats: GGUF (llama.cpp), GPTQ, AWQ, ExLlamaV2.
- Cache Storage
- Key-Value cache: structure that stores vectors of key-value pairs from the attention layer for already processed tokens, avoiding recalculation at each step but occupying VRAM proportional to context length. For long contexts (32k-128k tokens), it can become the main consumer of VRAM.
- Parameters (B)
- Number of model weights, typically indicated in billions (B). A 7B model has 7 billion parameters. The number of parameters is the primary determinant of quality and hardware requirements.
- Cost Multiplier for Overheads
- Multiplier factor estimating additional VRAM based on parameters only. Includes: KV cache, intermediate activations (forward pass), framework buffer (Ollama, vLLM). Defaults to 1.2 (20%) for medium contexts; increase to 1.5–2.0 for long contexts.
- GPU Tier
- Classification of GPU hardware by VRAM required: consumer (<8 GB: RTX 3080/4080), prosumer (≤16 GB: RTX 3090/4090), professional (≤80 GB: A100/H100), datacenter (>80 GB: cluster multi-GPU).
Do you need a custom analysis?
This tool is free and informative. For in-depth analysis with AI on-prem - private data, zero cloud - contact Federico.