How Much VRAM Do You Need for a 70B LLM? GPU Memory Requirements and Quantization Option

If you are planning to run a large language model on your own hardware, one question comes up before anything else: how much VRAM do you need for a 70B model? The answer depends on the precision you use, the context length you want and the software you run it on. A 70B model can need anywhere from about 24 GB to well over 150 GB of GPU memory.

This guide breaks it all down in plain language. You will learn the exact memory needs for FP16, 8-bit and 4-bit versions, how the KV cache quietly eats extra memory, which GPU setups actually work, what speed to expect and how much memory fine-tuning requires. By the end, you will know exactly what hardware fits your budget and your goals.

The Short Answer

What “70B” Really Means

The “B” stands for billions of parameters. A 70B model has roughly 70 billion learned values (weights) that must all sit in memory while the model generates text. Popular models in this size class include Llama 3.3 70B, Llama 3.1 70B, Qwen 72B variants and distilled reasoning models built on 70B bases.

The architecture matters too. Most of these are dense models, which means every parameter is stored in memory and used. Mixture-of-experts (MoE) models are different: they have a large total parameter count but activate only part of it per token. Even so, you still need enough memory to hold all the weights, so total parameters, not active parameters, decide your VRAM needs.

The Simple VRAM Formula

You can estimate memory for any model with one easy rule:

Bytes per parameter depends on precision: 2 bytes for FP16 or BF16, 1 byte for 8-bit and 0.5 bytes for 4-bit. The 1.2 multiplier is a safety margin for CUDA overhead and a modest context window.

Real quantized files use slightly more than the idealized 0.5 bytes, because formats like Q4_K_M keep some layers at higher precision. That is why a 4-bit 70B file is closer to 42 GB than 35 GB, and why the practical number to plan for is around 46 to 48 GB.

VRAM Needed at Each Precision (FP16, 8-bit, 4-bit)

FP16 / BF16: About 140 GB for Weights

This is the full-quality version most models are released in. The weights alone take about 140 GB, and once you add the KV cache and overhead, you should plan for 150 to 170 GB. That means two 80 GB GPUs at minimum, with limited room for long contexts, or four 48 GB cards for a more comfortable setup. FP16 is mostly used in production serving and research where maximum fidelity matters.

FP8 and INT8: About 70 to 75 GB for Weights

Eight-bit precision cuts memory in half with very little loss in quality, which makes it the sweet spot for professional deployments. A single 96 GB workstation GPU can hold a 70B FP8 model with room to spare for context. A single 80 GB card can load it too, but with very little headroom left for the KV cache.

4-bit (Q4_K_M, AWQ, GPTQ): About 40 to 42 GB for Weights

Four-bit quantization is the most popular way to run 70B models locally. Quality loss is small for most chat, writing and coding tasks, while memory drops to roughly a quarter of FP16. In practice you need around 46 to 48 GB of total memory, which is why 48 GB GPUs and dual 24 GB setups became so popular for this model size.

3-bit and 2-bit: Fits Small Cards, Costs Quality

Q3 variants (about 34 GB) and Q2 variants (about 26 GB) let a 70B model squeeze onto smaller hardware, but you will notice weaker reasoning, more mistakes and less reliable instruction following. A 4-bit version of a smaller model, such as a 32B model, often beats a 2-bit 70B model in quality, so consider that trade-off before going too low.

The Hidden Cost: KV Cache and Context Length

Weights are only part of the story. While generating text, the model stores attention data for every token in the conversation. This is called the KV cache, and it grows with context length.

For a Llama-style 70B model with 80 layers, 8 key-value heads and a head size of 128, each token needs about 0.33 MB in FP16. The formula is:

At 8K tokens the cache is a manageable 2.7 GB, but at 32K it grows to about 10.7 GB and at 128K it reaches roughly 43 GB for a single conversation. Storing the cache in FP8 cuts those numbers in half. Serving several users at once multiplies the cost, because each active conversation needs its own cache.

Other Overhead You Should Plan For

• CUDA context and libraries: Expect 1 to 2 GB used before the model even loads.
• Activations and temporary buffers: These grow with batch size and prompt length, especially during the first processing of a long prompt.
• Inference engine reserves: Servers like vLLM pre-allocate a fixed share of GPU memory (often around 90 percent) for the model plus cache, so they may show “full” memory by design.
• Multi-GPU splitting: Splitting a model across cards adds some duplicated buffers and communication overhead on each GPU.
• Your desktop: If the same GPU drives your monitor, the operating system can use another 0.5 to 1.5 GB.

Which GPU Setups Can Run a 70B Model?

The chart below summarizes which configurations work at each precision with a typical 8K context. Green means comfortable, amber means it fits but with tight limits, and red means it will not fit in VRAM alone.

24 GB (RTX 3090 or RTX 4090)

A single 24 GB card cannot hold a normal 4-bit 70B model. You can run an extreme 2-bit variant such as IQ2_XS with a small context, or split the model between GPU and system RAM. Both options involve compromises in quality or speed. Two of these cards together (48 GB) are a far better choice.

32 GB (RTX 5090)

The RTX 5090 offers 32 GB of GDDR7 memory and very high bandwidth, but 32 GB is still short of a 4-bit 70B model. It is excellent for 32B-class models and can handle a 2-bit 70B with a short context. For 70B at 4-bit, you need two of them (64 GB total).

48 GB (Dual 24 GB or a Single 48 GB Workstation Card)

This is the classic entry point for 70B at 4-bit. A single 48 GB card, such as the RTX 6000 Ada or A6000, loads the model cleanly with a moderate context. Two used 24 GB cards work as well, although the split adds overhead and leaves less room for long conversations.

64 GB (Dual RTX 5090)

Two RTX 5090 cards give you 64 GB and plenty of bandwidth. A 4-bit 70B model fits with around 15 GB left for context, making this a strong high-end consumer option. Remember the power draw: each card can pull around 575 W.

80 GB (A100 or H100)

A single 80 GB data center card runs 4-bit 70B models with a large amount of headroom for long contexts and batching. It can also load an FP8 version, but the fit is tight because the weights alone are about 70 GB.

96 GB (RTX PRO 6000 Blackwell)

The RTX PRO 6000 Blackwell offers 96 GB of GDDR7 memory on a single card, so it is the simplest way to run a 70B model in 8-bit or in 4-bit with very long context. No model splitting, no multi-GPU headaches. The trade-off is price, since it costs many times more than a consumer card.

160 GB and Above (Multiple 80 GB GPUs)

Running FP16 requires at least two 80 GB GPUs, and that leaves only around 20 GB for the cache, which is fine for moderate contexts. For long contexts or many simultaneous users, four 80 GB GPUs or newer high-memory accelerators with more than 140 GB each are the safer path.

Apple Silicon and CPU Offloading

Apple Silicon (Unified Memory)

Apple chips share one memory pool between CPU and GPU, which makes high-memory Macs attractive for large models. A Mac with 96 GB or 128 GB of unified memory runs a 4-bit 70B model comfortably. A 64 GB Mac is tight, because macOS limits how much memory the GPU can use by default, usually around three quarters of the total. Apple machines are quieter and more efficient than multi-GPU PCs, but they generate text more slowly than top NVIDIA cards.

CPU and GPU Offloading

Tools such as llama.cpp can keep some layers on the GPU and push the rest into system RAM. This lets a 24 GB card run a 4-bit 70B model, but speed falls sharply because system memory is far slower than VRAM. Expect only a few tokens per second. You also need plenty of RAM, typically 64 GB or more, so the offloaded layers and your operating system both fit.

Speed: Why Memory Bandwidth Matters

Generating each token requires reading essentially the entire model from memory. That makes text generation limited by memory bandwidth rather than raw compute power. A rough upper limit is:

These are best-case ceilings. Real-world speed is usually noticeably lower because of overhead, software efficiency and context length. Still, the table shows why a card with more bandwidth feels faster even when two cards have the same VRAM.

VRAM Needed to Fine-Tune a 70B Model

Training needs far more memory than inference because the GPU must also store gradients, optimizer states and activations.

• QLoRA (4-bit base model): Roughly 48 GB or more, depending on sequence length and batch size. This makes 70B fine-tuning possible on a single 48 GB card or on dual 24 GB GPUs with the right tooling.
• LoRA (FP16 base model): Around 160 to 200 GB, because the full-precision weights must stay loaded. Think two to three 80 GB GPUs.
• Full fine-tuning: About 16 bytes per parameter with AdamW and mixed precision, which is roughly 1.1 TB before activations. In practice this means a cluster of 16 or more 80 GB GPUs.

For most people, QLoRA is the realistic route. It gives strong results for style, domain knowledge and task-specific behavior at a tiny fraction of the hardware cost.

How to Reduce VRAM Usage

1. Choose a better quantization: Q4_K_M or an AWQ 4-bit build is usually the best balance of size and quality.
2. Quantize the KV cache: FP8 or INT8 cache halves the memory used by long contexts with minimal quality impact on most tasks.
3. Limit context length: Set the maximum context to what you truly need rather than the model’s full 128K limit.
4. Use efficient attention: Features such as FlashAttention and paged KV caching reduce wasted memory.
5. Lower the batch size: Fewer simultaneous requests means less cache and fewer buffers.
6. Offload sparingly: Moving a few layers to system RAM can rescue a setup that is just slightly short on VRAM.
7. Consider a smaller model: Modern 30B to 32B models are surprisingly capable and can run on a single 24 or 32 GB card.

Multi-GPU Tips and Common Mistakes

• Use a GPU count that divides evenly: Tensor parallelism in engines like vLLM usually works best with 2, 4 or 8 GPUs, because the model’s 64 attention heads must split evenly. Three GPUs often will not work.
• Match your GPUs: Mixing very different cards wastes memory, because the slowest or smallest card limits the setup.
• Check PCIe lanes: Consumer motherboards may run a second card at reduced speed, which slows tensor-parallel setups.
• Plan for power and cooling: Two high-end cards can need a 1,600 W power supply and a case with serious airflow.
• Do not forget the context: A model that “fits” at load time can still crash when a long prompt arrives. Leave at least 10 percent headroom.
• Do not trust weights-only numbers: A “42 GB model” does not run in 42 GB of VRAM.

What Should You Choose?

Frequently Asked Questions

How much VRAM do I need to run a 70B model?

You need about 46 to 48 GB for a 4-bit version, 75 to 80 GB for 8-bit and around 150 GB for FP16, all with a typical 8K context. Longer context lengths add more.

Can I run a 70B model on a 24 GB GPU?

Not at normal 4-bit quality in VRAM alone. You can use a heavily compressed 2-bit variant with a small context, or offload part of the model to system RAM, which makes it much slower.

Can I run a 70B model on 16 GB of VRAM?

Technically yes, with heavy CPU offloading, but it will be very slow, typically only a few tokens per second or less. A smaller model is a far better experience on 16 GB.

Is 48 GB of VRAM enough for a 70B model?

Yes for a 4-bit model with a moderate context window. It will not handle FP16 or 8-bit, and very long contexts may not fit.

Are two RTX 3090 or 4090 cards good for 70B?

Yes. Two 24 GB cards give you 48 GB, which runs a 4-bit 70B model well. It is one of the most cost-effective local setups, though the second card adds noise, heat and power use.

How much VRAM does a 70B model need with a 128K context?

With a 4-bit model, expect roughly 90 GB, because the FP16 cache alone is about 43 GB. With an FP8 cache it drops to around 70 GB. With FP16 weights, you will need several 80 GB GPUs.

Is 4-bit quantization good enough?

For most chat, writing, summarization and coding tasks, yes. Q4_K_M and similar methods keep most of the original quality. Tasks that need precise math or long, careful reasoning may benefit from 6-bit or 8-bit.

Why does my 70B model run out of memory even though it should fit?

Usually the KV cache, a long prompt, a large batch size or framework overhead pushes memory past the limit. Lower the context length, quantize the cache or reduce the batch size.

Can a Mac run a 70B model?

Yes. A Mac with 96 GB or 128 GB of unified memory handles a 4-bit 70B model comfortably, though generation is slower than on a high-end NVIDIA GPU.

Do MoE models need less VRAM than dense 70B models?

Not necessarily. MoE models run faster because they activate fewer parameters per token, but you still need enough memory to store all of the experts.

Final Thoughts

So, how much VRAM for a 70B model? The most useful numbers to remember are about 48 GB for 4-bit, 80 to 96 GB for 8-bit and 150 GB or more for FP16. Add extra memory for long contexts, many users or fine-tuning, and always leave headroom so your setup does not crash on a big prompt.

For most enthusiasts, a 48 GB to 96 GB setup running a 4-bit model delivers the best balance of quality, speed and cost. If you only need 70B occasionally, renting a cloud GPU is a smart way to test before you invest in hardware. Whatever you choose, use the formula in this guide to check the numbers before you buy.

Note: All figures are approximate. Actual memory use varies by model architecture, quantization format, inference software and settings. Always test with your own workload.