Llama 3.1 405B Instruct hardware requirements
Needs at least 4 GPUs at BF16 (Instinct MI355X OAM). For 1,000 tokens/s at 32k context you need 176 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $88.00/hr.
Top options for Llama 3.1 405B Instruct
The strongest part each vendor offers for Llama 3.1 405B Instruct at BF16 with 32,768 tokens per user, rentable parts first. Each card is one replica, the fewest accelerators that hold the model, and the line it draws on the graph below. Select a card to plot it.
Tokens per second as users grow
Each line is one replica, the fewest accelerators that hold the model, serving more users at once. It climbs while memory bandwidth is the limit, flattens once compute is, and stops where memory runs out at 32,768 tokens per user. Hover a line to bring it forward.
8 x GB200 NVL72 GPU 186GB: peaks at 288 tokens per second when memory fills at 36 users.4 x Instinct MI355X OAM: peaks at 130 tokens per second when memory fills at 17 users.8 x Ascend 910C: peaks at 72.2 tokens per second when memory fills at 10 users.16 x Gaudi 2 96GB: peaks at 100 tokens per second when memory fills at 38 users.8 x TPU v7 192GB: peaks at 269 tokens per second when memory fills at 39 users.
Memory fills at: 8 x GB200 NVL72 GPU 186GB 36 users, 4 x Instinct MI355X OAM 17 users, 8 x Ascend 910C 10 users, 16 x Gaudi 2 96GB 38 users, 8 x TPU v7 192GB 39 users. Roofline estimates at BF16.
Bare minimum
- Tokens/s
- 26.2
- Per hour
- $23.96
- Per million tokens
- $254
- Memory used
- 73%
- Tokens/s
- 19.5
- Per hour
- $8.00
- Per million tokens
- $114
- Memory used
- 55%
- Tokens/s
- 93.3
- Per hour
- $8.00
- Per million tokens
- $23.81
- Memory used
- 92%
Configurations
Every datacenter GPU in the catalog, sized for these settings. Change precision, context and concurrency and the whole page follows. Each point is one replica: the smallest tensor-parallel group of that GPU that holds the model.
8 x Instinct MI355X OAM: 284 tokens/s per replica at $47.92/hr.8 x GB300 NVL72 GPU 288GB: 282 tokens/s per replica at $99.36/hr.8 x GB200 NVL72 GPU 186GB: 282 tokens/s per replica at $84.00/hr.8 x Instinct MI350X OAM: 267 tokens/s per replica at $49.28/hr.8 x B300 SXM 262GB: 262 tokens/s per replica at $39.92/hr.8 x B200 SXM 180GB: 259 tokens/s per replica at $32.72/hr.8 x Instinct MI325X OAM: 162 tokens/s per replica at $30.40/hr.8 x Instinct MI300X 192GB: 157 tokens/s per replica at $19.12/hr.16 x H200 SXM 141GB: 219 tokens/s per replica at $39.68/hr.16 x H200 NVL 141GB: 191 tokens/s per replica at $57.82/hr.16 x H100 NVL 94GB: 183 tokens/s per replica at $51.04/hr.16 x RTX PRO 6000 Blackwell Workstation Edition: 103 tokens/s per replica at $28.80/hr.16 x RTX PRO 6000 Blackwell Server Edition: 99.6 tokens/s per replica at $11.04/hr.16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition: 93.3 tokens/s per replica at $8.00/hr.16 x Gaudi 2 96GB: 98.6 tokens/s per replica at $17.60/hr.
Priced configurations only; 23 more fit without a live price and appear in the table. Marker size grows with the tensor-parallel width. Roofline estimates.
| Rent | |||
|---|---|---|---|
| 4 | 331 tok/s | Specs → | |
| 8 | 515 tok/s | Specs → | |
TPU 8t announced | 8 | ~307 tok/s | Specs → |
TPU 8i announced | 8 | ~290 tok/s | Specs → |
| 8 | 284 tok/s | Rent → | |
| 8 | 282 tok/s | Rent → | |
| 8 | 282 tok/s | Rent → | |
| 8 | 267 tok/s | Rent → | |
| 8 | 260 tok/s | Specs → | |
Instinct MI430X OAM announced | 4 | ~172 tok/s | Specs → |
| 8 | 262 tok/s | Rent → | |
| 8 | 259 tok/s | Rent → | |
| 8 | 270 tok/s | Specs → | |
| 8 | 217 tok/s | Specs → | |
| 8 | ~237 tok/s | Specs → | |
| 8 | 162 tok/s | Rent → | |
| 8 | 157 tok/s | Rent → | |
| 8 | 140 tok/s | Specs → | |
H200 SXM 141GB 2 nodes | 16 | 219 tok/s | Rent → |
Data Center GPU Max 1550 128GB 2 nodes | 16 | 176 tok/s | Specs → |
H200 NVL 141GB 2 nodes | 16 | ~191 tok/s | Rent → |
H100 NVL 94GB 2 nodes | 16 | ~183 tok/s | Rent → |
| 16 | 169 tok/s | Specs → | |
| 16 | 130 tok/s | Specs → | |
| 16 | 103 tok/s | Rent → | |
| 16 | 99.6 tok/s | Specs → | |
| 16 | 99.6 tok/s | Rent → | |
Gaudi 3 128GB 2 nodes | 16 | 295 tok/s | Specs → |
TPU v5p 95GB 2 nodes | 16 | 106 tok/s | Specs → |
| 16 | 93.3 tok/s | Rent → | |
| 16 | ~59.8 tok/s | Specs → | |
Gaudi 2 96GB 2 nodes | 16 | 98.6 tok/s | Rent → |
| 16 | 85.4 tok/s | Specs → | |
MI250X 128GB 2 nodes | 16 | 93.4 tok/s | Specs → |
Instinct MI250 128GB 2 nodes | 16 | 89.0 tok/s | Specs → |
H20 141GB HBM3e 2 nodes | 16 | 39.7 tok/s | Specs → |
H20 96GB 2 nodes | 16 | 39.7 tok/s | Specs → |
| 16 | 27.1 tok/s | Specs → |
By vendor
At BF16 the NVIDIA pick is 16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at about 93.3 tokens/s for $8.00/hr; the AMD pick is 8 x Instinct MI300X 192GB at about 157 tokens/s for $19.12/hr. Per rental dollar NVIDIA delivers 1.42x the tokens of AMD here.
NVIDIA: 11.7 tokens/s per dollar, 19.4 tokens/s per kW (16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition).AMD: 8.2 tokens/s per dollar, 26.2 tokens/s per kW (8 x Instinct MI300X 192GB).Huawei: no live price, 17.6 tokens/s per kW (16 x Ascend 910C).Intel: 5.6 tokens/s per dollar, 10.3 tokens/s per kW (16 x Gaudi 2 96GB).Google: no live price, no power figure (8 x TPU v7 192GB).Other: no live price, 24.9 tokens/s per kW (16 x Cloud AI 100 Ultra).
Fleet what-if
Compare whole fleets at these settings: a thousand of one part against a hundred of another. Each fleet splits into replicas of its tensor-parallel width; GPUs left over sit idle.
| Fleet | Tokens/s |
|---|---|
1,008 x H200 SXM 141GB | 14k tok/s |
104 x GB200 NVL72 GPU 186GB | 3.7k tok/s |
1,008 x H200 SXM 141GB: 14k tokens/s, $2,500/hr, 706 kW.104 x GB200 NVL72 GPU 186GB: 3.7k tokens/s, $1,092/hr, no power figure.
H200 SXM 141GB: 14k tokens/s at 1008 GPUs.GB200 NVL72 GPU 186GB: 36k tokens/s at 1008 GPUs.
Size Llama 3.1 405B Instruct for a tokens per second target
The calculator starts from your traffic instead of a fleet: set a target rate and read the replica count for every GPU.
Frequently asked questions
How much GPU memory does Llama 3.1 405B Instruct need?
The weights take 812 GB at the native BF16 precision (812 GB at BF16, 406 GB at FP8, 228 GB at INT4). Each concurrent stream adds 504 KB of KV cache per token: 2.1 GB at 4,096 tokens and 16.9 GB at 32,768 tokens.
What is the bare minimum to run Llama 3.1 405B Instruct?
4 x Instinct MI355X OAM at BF16 with 4,096 tokens of context and one stream. The cheapest live rental for that footprint is 16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $8.00 per hour.
How many H100s do you need to run Llama 3.1 405B Instruct?
16 H100s at BF16 for 4,096 tokens of context and one stream. For 32,768 tokens and 32 concurrent streams the tensor-parallel group is more than sixteen H100s.
How much does Llama 3.1 405B Instruct cost per million tokens on an H100?
No live H100 rental price is on file right now, so the cost per token cannot be quoted. The configuration table lists every part that has one.
What precision does Llama 3.1 405B Instruct ship in?
The published checkpoint is BF16. FP8 and INT4 figures describe post-training quantisations that halve and quarter the footprint at a small accuracy cost.
All figures are roofline estimates from the model's config.json and each GPU's published memory, bandwidth and tensor-core peak, with fixed efficiency factors. Prices are the cheapest live on-demand listing per GPU when the site has one. They are a planning floor; a tuned serving stack can do better. The Meta repo is gated; geometry read from the RedHatAI FP8 copy, which keeps the 8 KV heads (two INT4 copies duplicate them to 16).