AWS Inferentia

Inferentia is the first machine learning chip AWS designed itself, announced at re:Invent 2018 and the part behind EC2 Inf1 instances from December 2019. Each chip is four NeuronCore-v1 cores rated by AWS at 64 TFLOPS of FP16 or BF16 and 128 TOPS of INT8, with 8 GB of DDR4 at 50 GB/s and a large pool of on-chip SRAM the compiler uses to keep weights close to the cores. It handles only FP16, BF16 and INT8; FP32, TF32 and the configurable FP8 arrived with Inferentia2, along with HBM in place of DDR4, which is where Inferentia2's ten-fold memory bandwidth gain comes from. NeuronLink-v1 at 32 GB/s per chip lets the sixteen chips of an inf1.24xlarge be pipelined so one model can be split across several of them. Inferentia is a cost-per-inference part, sold only as EC2 capacity, and AWS publishes no power figure, process node or die size for it.