Will it run?
Pick your card and a model size. This works out the VRAM it needs, whether it fits, and roughly how fast it will produce tokens.
- Weights
- —
- KV cache
- —
- Overhead
- —
- Total VRAM
- —
- Decode speed
- —
How these numbers are worked out
Weights are the parameter count times the effective bytes-per-weight of the quantisation. Those bytes-per-weight figures are measured, not assumed: each is a real GGUF file size for Llama-3-8B divided by its 8.03 billion parameters. Q4_K_M works out at 0.613, which is why a 4-bit 8B model is about 4.9 GB on disk and not the 5.6 GB a round "4 bits plus overhead" guess would give you. Those figures hold across model families, not just Llama: Mistral-7B gives 0.604 and Qwen2.5-32B gives 0.607, across 4.5× the parameters and 4.75× the vocabulary.
A note on units. Everything here is in GiB (1024³ bytes), because that is what graphics cards are actually specified in — a "12GB" card holds 12 GiB. Model repositories usually quote file sizes in decimal GB, so the same 8B Q4_K_M file is 4.92 GB there and 4.58 GiB here. Same file, two conventions.
KV cache grows with context length and with the model's layer count — not with its parameter count. Llama-3 has 32 layers at 8B and 80 at 70B: 2.5× the cache for 8.75× the parameters. Estimating it linearly from parameters is 50% low on an 8B and 75% high on a 70B, so this scales with the cube root instead, calibrated against real Llama-3 geometry. It assumes an fp16 cache, which is what most people run even with quantised weights. This is the part people forget, and it is why a model that fits at 4K stops fitting at 128K. Overhead covers activations and the runtime's own allocations.
Why the attention setting matters more than it looks. Llama-3 is grouped-query: 8 key/value heads however wide the model gets. Older multi-head models keep one KV head per attention head — 32 of them at 7B, 40 at 13B — so their cache is four to five times bigger for the same context. That is not a rounding error, it is the difference between fitting on a 12 GiB card and not: a Llama-2-7B at 8K needs about 4 GiB of cache where a Llama-3-8B needs 1 GiB. Nearly everything released since 2024 is grouped-query, which is the default here; switch it if you are running a Llama-2-era model. The multi-head figures check out against exact geometry to within 2%.
Decode speed is mostly memory-bound: to a first approximation, producing each token reads the weights and the live cache once, so tokens per second is roughly your card's memory bandwidth divided by the bytes read, times an efficiency factor of 0.64. That is a rough model rather than a description of what a runtime really does — flash attention, a quantised cache, batching, offload and how busy the card already is all move it. The 0.64 is fitted to two measurements on an RTX 3060 (360 GB/s): a 3B at FP16 and 2K context, which measures about 38 tokens/sec against 37 estimated here, and an 8B at Q4_K_M and 8K context, which benchmarks at roughly 42 against 38 here.
The speed figure is the slow end, not an average. It counts the cache at the full context you selected, i.e. a conversation that has already filled up. Decoding only re-reads what has actually been written, so a short prompt at 128K starts far faster and slows towards this number as it fills. Read it as the floor you will converge on, which is why it is labelled with the context it assumes.
These are estimates. Real usage depends on the runtime, the exact architecture — grouped-query attention changes the cache maths considerably — batch size, and how much VRAM your desktop is already using. Treat the fit thresholds as guidance, not a promise: "fits" means the estimate lands under 72% of your VRAM, which leaves room for the things this page cannot see.