Paper on AFM-server

Specifications

Additional Information

"We train AFM-server from scratch for 6.3T tokens on 8192 TPUv4 chips, using a sequence length of 4096 and a batch-size of 4096 sequences." "The AFM models are pre-trained on v4 and v5p Cloud TPU clusters" "AFM-server was trained on 8192 TPUv4 chips provisioned as 8 × 1024 chip slices, where slices are connected together by the data-center network (DCN)"

Cloud TPU clusters implies they trained on Google infrastructure