Nemotron 3 Ultra 550B-A55B hardware requirements
Needs at least 8 GPUs at BF16 (Instinct MI355X OAM). For 1,000 tokens/s at 32k context you need 32 x H100 SXM5 80GB at $58.56/hr or 48 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $24.00/hr.
Top options for Nemotron 3 Ultra 550B-A55B
The strongest part each vendor offers for Nemotron 3 Ultra 550B-A55B at BF16 with 32,768 tokens per user, rentable parts first. Each card is one replica, the fewest accelerators that hold the model, and the line it draws on the graph below. Select a card to plot it.
Tokens per second as users grow
Each line is one replica, the fewest accelerators that hold the model, serving more users at once. It climbs while memory bandwidth is the limit, flattens once compute is, and stops where memory runs out at 32,768 tokens per user. Hover a line to bring it forward.
8 x GB300 NVL72 GPU 288GB: peaks at 2.6k tokens per second and still has memory at 1024 users.8 x Instinct MI355X OAM: peaks at 2.6k tokens per second and still has memory at 1024 users.16 x Ascend 910C: peaks at 1.5k tokens per second and still has memory at 1024 users.16 x Gaudi 2 96GB: peaks at 787 tokens per second when memory fills at 426 users.8 x TPU v7 192GB: peaks at 2.2k tokens per second when memory fills at 437 users.
Memory fills at: 8 x GB300 NVL72 GPU 288GB beyond 1,024 users, 8 x Instinct MI355X OAM beyond 1,024 users, 16 x Ascend 910C beyond 1,024 users, 16 x Gaudi 2 96GB 426 users, 8 x TPU v7 192GB 437 users. Roofline estimates at BF16.
Bare minimum
- Tokens/s
- 364
- Per hour
- $47.92
- Per million tokens
- $36.54
- Memory used
- 51%
- Tokens/s
- 143
- Per hour
- $8.00
- Per million tokens
- $15.49
- Memory used
- 76%
- Tokens/s
- 355
- Per hour
- $8.00
- Per million tokens
- $6.26
- Memory used
- 78%
Configurations
Every datacenter GPU in the catalog, sized for these settings. Change precision, context and concurrency and the whole page follows. Each point is one replica: the smallest tensor-parallel group of that GPU that holds the model.
8 x Instinct MI355X OAM: 983 tokens/s per replica at $47.92/hr.8 x GB300 NVL72 GPU 288GB: 981 tokens/s per replica at $99.36/hr.8 x GB200 NVL72 GPU 186GB: 981 tokens/s per replica at $84.00/hr.8 x Instinct MI350X OAM: 954 tokens/s per replica at $49.28/hr.8 x B300 SXM 262GB: 946 tokens/s per replica at $39.92/hr.8 x B200 SXM 180GB: 923 tokens/s per replica at $32.72/hr.8 x Instinct MI325X OAM: 643 tokens/s per replica at $30.40/hr.8 x Instinct MI300X 192GB: 597 tokens/s per replica at $19.12/hr.16 x H200 SXM 141GB: 886 tokens/s per replica at $39.68/hr.16 x H100 SXM5 80GB: 711 tokens/s per replica at $29.28/hr.16 x H200 NVL 141GB: 821 tokens/s per replica at $57.82/hr.16 x H100 NVL 94GB: 732 tokens/s per replica at $51.04/hr.16 x H100 PCIe 80GB: 459 tokens/s per replica at $40.00/hr.16 x RTX PRO 6000 Blackwell Workstation Edition: 374 tokens/s per replica at $28.80/hr.16 x RTX PRO 6000 Blackwell Server Edition: 346 tokens/s per replica at $11.04/hr.16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition: 355 tokens/s per replica at $8.00/hr.16 x Gaudi 2 96GB: 422 tokens/s per replica at $17.60/hr.16 x A100 PCIe 80GB: 319 tokens/s per replica at $21.60/hr.16 x A100 SXM4 80GB: 327 tokens/s per replica at $22.40/hr.
Priced configurations only; 27 more fit without a live price and appear in the table. Marker size grows with the tensor-parallel width. Roofline estimates.
| Rent | |||
|---|---|---|---|
| 4 | 1.3k tok/s | Specs → | |
| 8 | 2.2k tok/s | Specs → | |
TPU 8t announced | 8 | ~907 tok/s | Specs → |
TPU 8i announced | 8 | ~1.0k tok/s | Specs → |
| 8 | 983 tok/s | Rent → | |
| 8 | 981 tok/s | Rent → | |
| 8 | 981 tok/s | Rent → | |
| 8 | 954 tok/s | Rent → | |
| 8 | 905 tok/s | Specs → | |
Instinct MI430X OAM announced | 4 | ~878 tok/s | Specs → |
| 8 | 946 tok/s | Rent → | |
| 8 | 923 tok/s | Rent → | |
| 8 | 1.2k tok/s | Specs → | |
| 8 | 858 tok/s | Specs → | |
| 8 | ~1.2k tok/s | Specs → | |
| 8 | 643 tok/s | Rent → | |
| 8 | 597 tok/s | Rent → | |
| 8 | 706 tok/s | Specs → | |
H200 SXM 141GB 2 nodes | 16 | 886 tok/s | Rent → |
H100 SXM5 80GB 2 nodes | 16 | 711 tok/s | Rent → |
Data Center GPU Max 1550 128GB 2 nodes | 16 | 660 tok/s | Specs → |
H200 NVL 141GB 2 nodes | 16 | ~821 tok/s | Rent → |
H100 NVL 94GB 2 nodes | 16 | ~732 tok/s | Rent → |
| 16 | 639 tok/s | Specs → | |
H100 PCIe 80GB 2 nodes | 16 | 459 tok/s | Rent → |
H800 PCIe 80GB 2 nodes | 16 | 459 tok/s | Specs → |
| 16 | 606 tok/s | Specs → | |
| 16 | 374 tok/s | Rent → | |
| 16 | 346 tok/s | Specs → | |
| 16 | ~346 tok/s | Specs → | |
| 16 | 346 tok/s | Rent → | |
Gaudi 3 128GB 2 nodes | 16 | 893 tok/s | Specs → |
TPU v5p 95GB 2 nodes | 16 | 462 tok/s | Specs → |
| 16 | 355 tok/s | Rent → | |
| 16 | ~149 tok/s | Specs → | |
Gaudi 2 96GB 2 nodes | 16 | 422 tok/s | Rent → |
| 16 | 300 tok/s | Specs → | |
MI250X 128GB 2 nodes | 16 | 455 tok/s | Specs → |
Instinct MI250 128GB 2 nodes | 16 | 441 tok/s | Specs → |
A800 PCIe 80GB 2 nodes | 16 | 319 tok/s | Specs → |
A800 SXM4 2 nodes | 16 | 327 tok/s | Specs → |
A100 PCIe 80GB 2 nodes | 16 | 319 tok/s | Rent → |
A100 SXM4 80GB 2 nodes | 16 | 327 tok/s | Rent → |
H20 141GB HBM3e 2 nodes | 16 | 249 tok/s | Specs → |
H20 96GB 2 nodes | 16 | 249 tok/s | Specs → |
| 16 | 91.0 tok/s | Specs → |
By vendor
At BF16 the NVIDIA pick is 16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at about 355 tokens/s for $8.00/hr; the AMD pick is 8 x Instinct MI300X 192GB at about 597 tokens/s for $19.12/hr. Per rental dollar NVIDIA delivers 1.42x the tokens of AMD here.
NVIDIA: 44.4 tokens/s per dollar, 74.0 tokens/s per kW (16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition).AMD: 31.2 tokens/s per dollar, 99.4 tokens/s per kW (8 x Instinct MI300X 192GB).Huawei: no live price, 66.6 tokens/s per kW (16 x Ascend 910C).Intel: 24.0 tokens/s per dollar, 43.9 tokens/s per kW (16 x Gaudi 2 96GB).Google: no live price, no power figure (8 x TPU v7 192GB).Other: no live price, no power figure (16 x MTT S5000).
Fleet what-if
Compare whole fleets at these settings: a thousand of one part against a hundred of another. Each fleet splits into replicas of its tensor-parallel width; GPUs left over sit idle.
| Fleet | Tokens/s |
|---|---|
1,008 x H200 SXM 141GB | 56k tok/s |
104 x GB200 NVL72 GPU 186GB | 13k tok/s |
1,008 x H200 SXM 141GB: 56k tokens/s, $2,500/hr, 706 kW.104 x GB200 NVL72 GPU 186GB: 13k tokens/s, $1,092/hr, no power figure.
H200 SXM 141GB: 56k tokens/s at 1008 GPUs.GB200 NVL72 GPU 186GB: 124k tokens/s at 1008 GPUs.
Size Nemotron 3 Ultra 550B-A55B for a tokens per second target
The calculator starts from your traffic instead of a fleet: set a target rate and read the replica count for every GPU.
Frequently asked questions
How much GPU memory does Nemotron 3 Ultra 550B-A55B need?
The weights take 1,121 GB at the native BF16 precision (1,121 GB at BF16, 561 GB at FP8, 315 GB at INT4). Each concurrent stream adds 12 KB of KV cache per token: 0.1 GB at 4,096 tokens and 0.4 GB at 32,768 tokens.
What is the bare minimum to run Nemotron 3 Ultra 550B-A55B?
8 x Instinct MI355X OAM at BF16 with 4,096 tokens of context and one stream. The cheapest live rental for that footprint is 16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $8.00 per hour.
How many H100s do you need to run Nemotron 3 Ultra 550B-A55B?
16 H100s at BF16 for 4,096 tokens of context and one stream. For 32,768 tokens and 32 concurrent streams the tensor-parallel group is 16 H100s, and 1,000 tokens per second takes 32 H100s across 2 replicas.
How much does Nemotron 3 Ultra 550B-A55B cost per million tokens on an H100?
About $11.44 per million output tokens at BF16, 32,768 tokens of context and 32 streams, using the cheapest live on-demand price of $1.83 per GPU-hour. The cheapest part per token is 16 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $6.26.
What precision does Nemotron 3 Ultra 550B-A55B ship in?
The published checkpoint is BF16. FP8 and INT4 figures describe post-training quantisations that halve and quarter the footprint at a small accuracy cost.
All figures are roofline estimates from the model's config.json and each GPU's published memory, bandwidth and tensor-core peak, with fixed efficiency factors. Prices are the cheapest live on-demand listing per GPU when the site has one. They are a planning floor; a tuned serving stack can do better. Hybrid Mamba-Transformer with latent-width MoE experts and a multi-token-prediction layer. 12 attention layers keep a per-token cache (12 KB per token); 48 Mamba-2 layers hold 408 MB of fixed FP32 state per sequence. NVIDIA's stated minimum is 8 B200, GB200, B300 or GB300, 8 H200 or 16 H100 at BF16, and 4 B200 or 8 H100 at NVFP4. Our memory floor keeps 10 percent of each GPU free and no serving headroom, so it asks for 16 H200 at BF16 and 2 B200 at NVFP4.