Open the datasheet for almost any datacenter GPU launched since 2020 and the FP8 line will carry two numbers. For the NVIDIA H100 SXM5 they are 1,979 and 3,958 TFLOPS. The second is exactly double the first, and the footnote that explains why is easy to miss.
This guide covers what that footnote means, which workloads can reach the bigger number, and which of the two you should put in a comparison.
Why the same GPU has two FP8 numbers
The bigger figure assumes structured sparsity, a hardware feature of the tensor cores that lets them skip part of a matrix multiply when the weights follow a fixed pattern of zeros.
The smaller figure is the dense rate: what the tensor cores deliver when every weight is a real value. It is the number a training run or an unmodified model will see.
Vendors print both because both are true. The sparse figure is the peak of the hardware. The dense figure is the peak of the hardware on the work most people actually give it.
What 2:4 structured sparsity actually does
The pattern is called 2:4. Take the weights of a layer in groups of four. In each group, at most two may be non-zero. The other two are stored as nothing more than a two-bit index that says which positions were kept.
Because the layout is fixed, the tensor core does not have to search for zeros. It reads the two surviving values and their indices, fetches only the matching activations, and skips the multiplies that would have produced zero. Half the arithmetic disappears, so the throughput figure doubles.
NVIDIA introduced the feature with the Ampere A100 in 2020. Every NVIDIA datacenter GPU since has carried it, and AMD’s CDNA parts from the MI300 series onwards publish sparse figures of their own.
Where the 2x comes from
The doubling is arithmetic, not a benchmark. A 2:4 pattern removes exactly half of the multiply-accumulate operations in a matrix multiply, so the peak rate is exactly twice the dense peak. That is why the sparse figure on a datasheet is always precisely 2x, never 1.8x or 2.3x.
When the sparse figure applies, and when it does not
The hardware only skips work that the model has already removed. To reach the sparse rate a model must be pruned to the 2:4 pattern, usually followed by a round of fine-tuning to recover accuracy, and then run through software that stores the weights in the compressed layout.
That puts the sparse figure out of reach for most work:
- Training runs dense. The weights are changing on every step, so there is no fixed pattern to exploit. Research into sparse training exists, but production training uses the dense rate.
- Inference on an unmodified checkpoint runs dense. Downloading a model and serving it does not make it sparse.
- Inference on a model pruned to 2:4 can use the sparse rate for the matrix multiplies in its linear layers. Attention, normalisation and everything else still run at dense or memory-bound speed.
Even in the third case, a model is rarely twice as fast end to end. The sparse rate applies to the share of runtime spent in the pruned matrix multiplies, and large-model inference is often limited by memory bandwidth rather than arithmetic.
Which precisions can have a sparse figure
Structured sparsity is a tensor-core feature, so it only exists for the tensor-core precisions: FP8, FP16, BF16, TF32, INT8 and, on newer parts, FP6 and FP4.
FP32 and FP64 vector throughput run on the ordinary CUDA or stream cores. There is no 2:4 mode there, so a datasheet never prints a sparse FP64 figure. If you see one, somebody has doubled a number they should not have.
The H100 SXM5 line shows the shape of a typical datasheet:
| Precision | Dense TFLOPS | Sparse TFLOPS |
|---|---|---|
| FP8 tensor core | 1,979 | 3,958 |
| FP16 tensor core | 990 | 1,979 |
| TF32 tensor core | 494 | 989 |
| FP32 vector | 67 | no figure |
| FP64 vector | 34 | no figure |
How vendors publish it
NVIDIA has printed the pair on every datacenter datasheet since Ampere, often as a single cell such as “3,958 | 1,979” with the sparse figure first, or with a footnote reading “with sparsity”. Older parts such as the Volta V100 have no structured sparsity, so they carry one number.
AMD’s MI300X datasheet gives the pair for its tensor-core precisions. The MI350 series brochures do the same for FP16, BF16, FP8 and INT8, then print “N/A” in the sparse column for MXFP6 and MXFP4. An empty cell on a vendor’s own table is data: it means the vendor does not claim a sparse rate at that precision, and nobody else should claim one for them.
Other vendors publish a single figure and leave the basis unstated. Some of those single figures are sparse peaks. Treat an unlabelled number with care and look for the dense rate in the footnotes before comparing it with anything.
How Flopper shows it
Every GPU page on this site leads with the dense figure and shows the sparse figure beside it when the vendor has published one. Where the hardware supports 2:4 but the vendor printed no sparse number, the table says so rather than doubling the dense rate for them. Where the part has no structured sparsity at all, the sparse column is marked as not applicable.
We never derive a sparse figure. If a vendor’s own document does not print it, the cell stays empty, and the methodology page explains the rule.
A checklist for reading a datasheet
- Find the footnote. “With sparsity”, “2:4”, “structured sparse” or a ”|” between two numbers all mean the larger figure is the sparse one.
- Compare dense with dense. A sparse FP8 figure from one vendor against a dense FP8 figure from another is a 2x error before you have started.
- Ask whether your model is pruned. If the answer is no, the dense number is your ceiling.
- Ignore any sparse FP32 or FP64 figure. The hardware has no such mode.
- Check the precision as well as the basis. Doubling from FP16 to FP8 and doubling for sparsity compound, so a headline “4 PFLOPS” can be a dense FP16 part at under 1 PFLOPS. The TFLOPS entry and how to calculate TFLOPS cover the arithmetic.
Two numbers on one line is not a vendor hiding something. It is the datasheet telling you which workload it is describing, as long as you read the footnote.