Transformer Engine

Hardware and software technology in Hopper architecture designed to accelerate Transformer models (GPT, BERT, etc.). Automatically selects optimal precision (FP8 or FP16) on a per-layer basis, providing up to 6x speedup for large language model training.