Existing Confirmed

Google TPUv5e

Google • November 8, 2023

Total GPUs

50.9K

H100 Equiv

10.1K

Power

14.8 MW

FP16

10.0K PF

Specifications

Primary GPU	Google TPU v5e
GPU Count	50.9K
Sector	Private

Additional Information

we used Multislice Training to run what we believe to be the world’s largest publicly disclosed LLM distributed training job (in terms of the number of chips used for training) on a compute cluster of 50,944 Cloud TPU v5e chips (spanning 199 Cloud TPU v5e pods) that is capable of achieving 10 exa-FLOPs (16-bit), or 20 exa-OPs (8-bit), of total peak performance