LLM GPU Calculator

How this is calculated

• Weights are parameters times bytes per parameter: 2 at BF16, 1 at FP8, 0.5625 at INT4 or FP4 including block scales. The figure is the full checkpoint, not the active subset.
• KV cache follows the model's config.json: layers times KV heads times head dimension, K and V, in 16-bit. Sliding-window layers are charged for the window only, MLA models for their latent, and linear-attention layers for a fixed state per stream.
• A GPU offers 90 percent of its memory to weights and cache, less 1 GB of workspace. Tensor parallel widens across 1, 2, 4, 8 and 16 GPUs until the pool fits.
• Decode is bandwidth-bound: every step streams the active weights and every stream's cache at 75 percent of peak bandwidth, with a compute floor at 60 percent of the dense tensor peak. FP8 weights compute at FP8 where the part has it; 4-bit weights compute at 16-bit unless the part has FP4 tensor cores.
• Replicas are the target rate divided by one replica's rate, rounded up. Total GPUs is replicas times the tensor-parallel width. Measured throughput from a tuned serving stack can beat these numbers; they are a planning floor, not a benchmark.