LLM GPU Calculator

How this is calculated

• Weights are parameters times bytes per parameter: 2 at BF16, 1 at FP8, 0.5625 at INT4 or FP4 including block scales. The figure is the full checkpoint, not the active subset.
• KV cache follows the model's config.json: layers times KV heads times head dimension, K and V, in 16-bit. Sliding-window layers are charged for the window only, MLA models for their latent, and linear-attention layers for a fixed state per stream.
• A GPU offers 90 percent of its memory to weights and cache, less 1 GB of workspace. Tensor parallel widens across 1, 2, 4, 8 and 16 GPUs until the pool fits.
• Prefill is compute-bound: two FLOPs per active parameter for every prompt token the prefix cache does not already hold, at 60 percent of the dense tensor peak. Prompt tokens are the context less the output per request; the cache hit rate is the share of them served from an earlier request (system prompts, shared documents, chat history). The attention term that grows with the square of the prompt is not counted, so very long prompts cost a little more than shown.
• Decode is bandwidth-bound: every step streams the weights the batch touches and every stream's cache at 75 percent of peak bandwidth, with a compute floor at 60 percent of the dense tensor peak. For a mixture of experts the weights touched grow with concurrency: a batch of n streams choosing k of E experts lands on E(1 - (1 - k/E)^n) distinct experts, so DeepSeek-class models at 32 streams read most of the expert pool, not the active parameters. FP8 weights compute at FP8 where the part has it; 4-bit weights compute at 16-bit unless the part has FP4 tensor cores.
• A request costs its prefill plus its output tokens' share of the decode steps it sits in, so a replica's rate is output tokens over that GPU time; "prefill" in the table marks parts where prefill is more than half of it. Time to first token is the prefill of one request on the fastest fitting part. Replicas are the target rate divided by one replica's rate, rounded up. Total GPUs is replicas times the tensor-parallel width. Tensor parallel is the only split modelled; expert parallel and disaggregated prefill can do better. Measured throughput from a tuned serving stack can beat these numbers; they are a planning floor, not a benchmark.