- Home
- Datacenters
- Paper on Megatron-Turing
Existing Confirmed
Paper on Megatron-Turing
Microsoft,NVIDIA • January 28, 2022
Total GPUs
4.5K
H100 Equiv
1.4K
Power
3.8 MW
FP16
1.4K PF
System Equivalents
This datacenter's 4.5K GPUs equal:
NVIDIA DGX A100
8 GPUs/system
560
NVIDIA HGX A100 (8-GPU)
8 GPUs/system
560
Specifications
| Primary GPU | NVIDIA A100 SXM4 80 GB |
| GPU Count | 4.5K |
| Sector | Private |
Additional Information
"Model training is done with mixed precision using 16-bit bfloat on NVIDIA’s Selene [2] supercomputer with 560 DGX A100 nodes. Each cluster node has 8 NVIDIA 80-GB A100 GPUs"