AWS Inferentia2

Inferentia2 is the inference half of the AWS Neuron line, the chip that sits under EC2 Inf2 while Trainium sits under Trn. Each chip is two NeuronCore-v2 cores over 32 GB of HBM at 820 GB/s, rated at 190 TFLOPS across FP16, BF16, cFP8 and TF32, 380 TOPS of INT8 and 47.5 TFLOPS of FP32. The generation is where AWS added FP32, TF32 and its configurable FP8 format to a line that had only done FP16, BF16 and INT8, and where it first put a chip-to-chip interconnect on an inference part: NeuronLink-v2 at 192 GB/s per chip, which is what lets a model larger than 32 GB be sharded across the twelve chips of an inf2.48xlarge instead of being confined to one. Against a contemporary GPU the compute is modest and the memory is small, and that is the point. Inferentia2 is sold as cost per token rather than peak throughput, and it is only available through EC2, never as a component. AWS publishes no power figure, no process node and no die size for any Neuron chip, so none is shown here.