Mistral AI Mistral Medium

Mistral Medium 3.5 128B hardware requirements

Fits on one Instinct MI355X OAM at FP8. For 1,000 tokens/s at 32k context you need 16 x H100 SXM5 80GB at $29.28/hr or 32 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $16.00/hr.

Parameters

128B

Active per token

All

dense model

KV cache per token

352 KB

16-bit cache

Native precision

FP8

Rent Model card

Top options for Mistral Medium 3.5 128B

The strongest part each vendor offers for Mistral Medium 3.5 128B at FP8 with 32,768 tokens per user, rentable parts first. Each card is one replica, the fewest accelerators that hold the model, and the line it draws on the graph below. Select a card to plot it.

Tokens per second as users grow

Each line is one replica, the fewest accelerators that hold the model, serving more users at once. It climbs while memory bandwidth is the limit, flattens once compute is, and stops where memory runs out at 32,768 tokens per user. Hover a line to bring it forward.

1 x GB300 NVL72 GPU 288GB 1 x Instinct MI355X OAM 2 x Ascend 950DT 2 x Gaudi 2 96GB 1 x TPU v7 192GB 8 x Inferentia2

1 x GB300 NVL72 GPU 288GB and 1 x Instinct MI355X OAM draw the same line: identical memory and bandwidth give identical curves.

Memory fills at: 1 x GB300 NVL72 GPU 288GB 12 users, 1 x Instinct MI355X OAM 12 users, 2 x Ascend 950DT 12 users, 2 x Gaudi 2 96GB 4 users, 1 x TPU v7 192GB 4 users, 8 x Inferentia2 9 users. Roofline estimates at FP8.

Bare minimum

Fewest GPUs

1 x Instinct MI355X OAM

FP8, 4,096 tokens, one stream

Tokens/s: 45.8
Per hour: $5.99
Per million tokens: $36.32
Memory used: 47%

Rent on Daytona

Cheapest per hour

2 x RTX PRO 6000 Blackwell Max-Q Workstation Edition

FP8, 4,096 tokens, one stream, live price

Tokens/s: 19.4
Per hour: $1.00
Per million tokens: $14.30
Memory used: 70%

Rent on RunPod

Cheapest per million tokens

8 x RTX PRO 6000 Blackwell Max-Q Workstation Edition

FP8, 32,768 tokens, 32 streams

Tokens/s: 252
Per hour: $4.00
Per million tokens: $4.42
Memory used: 69%

Rent on RunPod

Configurations

Every datacenter GPU in the catalog, sized for these settings. Change precision, context and concurrency and the whole page follows. Each point is one replica: the smallest tensor-parallel group of that GPU that holds the model.

Precision Context Streams

Vendor

Priced only

Priced configurations only; 42 more fit without a live price and appear in the table. Marker size grows with the tensor-parallel width. Roofline estimates.

#									Rent
1	Instinct MI455X	2	61 %	1.2k tok/s	65.7 tok/s	—	—	—	Specs →
2	Rubin SXM	2	91 %	1.1k tok/s	62.0 tok/s	—	—	—	Specs →
3	TPU 8t announced	4	61 %	~481 tok/s	34.9 tok/s	—	—	—	Specs →
4	TPU 8i announced	2	91 %	~245 tok/s	24.2 tok/s	—	—	—	Specs →
5	Instinct MI355X OAM	2	91 %	359 tok/s	22.5 tok/s	$11.98	$9.28	2.8 kW	Rent →
6	GB300 NVL72 GPU 288GB	2	91 %	357 tok/s	22.5 tok/s	$24.84	$19.30	—	Rent →
7	GB200 NVL72 GPU 186GB	4	71 %	677 tok/s	42.7 tok/s	$42.00	$17.23	—	Rent →
8	Instinct MI350X OAM	2	91 %	343 tok/s	22.5 tok/s	$12.32	$9.98	2.0 kW	Rent →
9	TPU v7 192GB	4	69 %	625 tok/s	39.4 tok/s	—	—	—	Specs →
10	Instinct MI430X OAM announced	2	61 %	~282 tok/s	65.7 tok/s	—	—	—	Specs →
11	B300 SXM 262GB	4	50 %	641 tok/s	42.7 tok/s	$19.96	$8.65	5.6 kW	Rent →
12	B200 SXM 180GB	4	73 %	630 tok/s	41.1 tok/s	$16.36	$7.22	4.0 kW	Rent →
13	Ascend 970 announced	2	91 %	395 tok/s	40.6 tok/s	—	—	—	Specs →
14	B100 SXM 192GB	4	69 %	557 tok/s	42.7 tok/s	—	—	2.8 kW	Specs →
15	Jalapeño announced	4	61 %	~678 tok/s	82.2 tok/s	—	—	2.8 kW	Specs →
16	Instinct MI325X OAM	4	51 %	417 tok/s	32.0 tok/s	$15.20	$10.13	4.0 kW	Rent →
17	Instinct MI300X 192GB	4	69 %	395 tok/s	28.3 tok/s	$9.56	$6.72	3.0 kW	Rent →
18	BR100 2 nodes	16	52 %	~457 tok/s	28.5 tok/s	—	—	8.8 kW	Specs →
19	Ascend 960 announced	2	91 %	214 tok/s	27.1 tok/s	—	—	—	Specs →
20	H200 SXM 141GB	4	94 %	322 tok/s	25.6 tok/s	$9.92	$8.55	2.8 kW	Rent →
21	H100 SXM5 80GB	8	83 %	520 tok/s	33.8 tok/s	$14.64	$7.81	5.6 kW	Rent →
22	Data Center GPU Max 1550 128GB	8	52 %	~303 tok/s	33.0 tok/s	—	—	4.8 kW	Specs →
23	H200 NVL 141GB	4	94 %	~290 tok/s	25.6 tok/s	$14.46	$13.85	2.4 kW	Rent →
24	H100 NVL 94GB	8	70 %	~506 tok/s	39.3 tok/s	$25.52	$14.00	3.2 kW	Rent →
25	Ascend 910C	8	52 %	~291 tok/s	32.3 tok/s	—	—	4.8 kW	Specs →
26	H100 PCIe 80GB	8	83 %	351 tok/s	20.2 tok/s	$20.00	$15.85	2.8 kW	Rent →
27	H800 PCIe 80GB	8	83 %	351 tok/s	20.2 tok/s	—	—	2.8 kW	Specs →
28	Ascend 950DT	4	92 %	197 tok/s	21.4 tok/s	—	—	—	Specs →
29	RTX PRO 6000 Blackwell Workstation Edition	8	69 %	271 tok/s	18.1 tok/s	$14.40	$14.74	4.8 kW	Rent →
30	Ascend 950PR	8	52 %	256 tok/s	16.1 tok/s	—	—	7.2 kW	Specs →
31	MTT S5000	8	83 %	~256 tok/s	16.1 tok/s	—	—	—	Specs →
32	RTX PRO 6000 Blackwell Server Edition	8	69 %	256 tok/s	16.1 tok/s	$5.52	$6.00	4.8 kW	Rent →
33	Gaudi 3 128GB	8	52 %	497 tok/s	37.3 tok/s	—	—	7.2 kW	Specs →
34	TPU v5p 95GB	8	70 %	185 tok/s	27.9 tok/s	—	—	3.6 kW	Specs →
35	RTX PRO 6000 Blackwell Max-Q Workstation Edition	8	69 %	252 tok/s	18.1 tok/s	$4.00	$4.42	2.4 kW	Rent →
36	Cloud AI 100 Ultra	8	52 %	~98.2 tok/s	5.5 tok/s	—	—	1.2 kW	Specs →
37	Gaudi 2 96GB	8	69 %	282 tok/s	24.7 tok/s	$8.80	$8.66	4.8 kW	Rent →
38	Atlas 350	8	59 %	214 tok/s	14.1 tok/s	—	—	4.8 kW	Specs →
39	Data Center GPU Max 1100 48GB 2 nodes	16	70 %	~244 tok/s	21.9 tok/s	—	—	4.8 kW	Specs →
40	Ascend 910B 2 nodes	16	52 %	~257 tok/s	28.5 tok/s	—	—	6.4 kW	Specs →
41	MI250X 128GB	8	52 %	~164 tok/s	33.1 tok/s	—	—	4.5 kW	Specs →
42	RTX 6000 Ada 48GB 2 nodes	16	70 %	~297 tok/s	17.1 tok/s	$15.52	$14.50	4.8 kW	Rent →
43	Instinct MI250 128GB	8	52 %	~156 tok/s	33.0 tok/s	—	—	4.0 kW	Specs →
44	L40S 48GB 2 nodes	16	70 %	281 tok/s	15.4 tok/s	$12.16	$12.01	5.6 kW	Rent →
45	A800 PCIe 80GB	8	83 %	~126 tok/s	19.5 tok/s	—	—	2.4 kW	Specs →
46	A800 SXM4	8	83 %	~128 tok/s	20.6 tok/s	—	—	3.2 kW	Specs →
47	A100 PCIe 80GB	8	83 %	~126 tok/s	19.5 tok/s	$10.80	$23.76	2.4 kW	Rent →
48	A100 SXM4 80GB	8	83 %	~128 tok/s	20.6 tok/s	$11.20	$24.39	3.2 kW	Rent →
49	A100 PCIe 40GB 2 nodes	16	84 %	~212 tok/s	27.7 tok/s	—	—	4.0 kW	Specs →
50	A100 SXM4 40GB 2 nodes	16	84 %	~212 tok/s	27.7 tok/s	$24.98	$32.68	6.4 kW	Rent →
51	RTX 5880 Ada 48GB 2 nodes	16	70 %	~260 tok/s	17.1 tok/s	—	—	4.6 kW	Specs →
52	RTX PRO 5000 Blackwell 48GB 2 nodes	16	70 %	~177 tok/s	23.9 tok/s	$15.36	$24.05	4.8 kW	Rent →
53	RTX PRO 5000 Blackwell 72GB	8	92 %	~101 tok/s	13.6 tok/s	—	—	2.4 kW	Specs →
54	L40 48GB 2 nodes	16	70 %	195 tok/s	15.4 tok/s	$13.12	$18.66	4.8 kW	Rent →
55	Instinct MI210 PCIe 2 nodes	16	52 %	~138 tok/s	29.2 tok/s	—	—	4.8 kW	Specs →
56	RTX A6000 48GB 2 nodes	16	70 %	~105 tok/s	13.7 tok/s	$8.00	$21.12	4.8 kW	Rent →
57	A40 48GB 2 nodes	16	70 %	~100 tok/s	12.4 tok/s	$9.44	$26.19	4.8 kW	Rent →
58	H20 141GB HBM3e	4	94 %	71.2 tok/s	21.4 tok/s	—	—	1.6 kW	Specs →
59	H20 96GB	8	69 %	134 tok/s	40.3 tok/s	—	—	3.2 kW	Specs →
60	Atlas 300I Duo 48GB 2 nodes	16	70 %	~81.4 tok/s	7.3 tok/s	—	—	2.4 kW	Specs →
61	Atlas 300I Duo 96GB	8	69 %	~46.1 tok/s	4.1 tok/s	—	—	1.2 kW	Specs →
62	ATOM-Max 2 nodes	16	52 %	~95.7 tok/s	18.2 tok/s	—	—	5.6 kW	Specs →
63	Radeon PRO W7900 Dual Slot 48GB 2 nodes	16	70 %	~90.0 tok/s	15.4 tok/s	—	—	4.7 kW	Specs →
64	Radeon PRO W7900 48GB 2 nodes	16	70 %	~90.0 tok/s	15.4 tok/s	—	—	4.7 kW	Specs →
65	L20 48GB 2 nodes	16	70 %	149 tok/s	15.4 tok/s	—	—	4.4 kW	Specs →
66	MLU370-X8 2 nodes	16	70 %	~69.0 tok/s	10.9 tok/s	—	—	4.0 kW	Specs →
67	Radeon PRO W7800 48GB 2 nodes	16	70 %	~69.5 tok/s	15.4 tok/s	—	—	4.2 kW	Specs →
68	Atlas 300V Pro 2 nodes	16	70 %	~40.8 tok/s	3.6 tok/s	—	—	—	Specs →

By vendor

At FP8 the NVIDIA pick is 8 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at about 252 tokens/s for $4.00/hr; the AMD pick is 4 x Instinct MI300X 192GB at about 395 tokens/s for $9.56/hr. Per rental dollar NVIDIA delivers 1.52x the tokens of AMD here.

NVIDIA 32 parts fit

Best value: 8 x RTX PRO 6000 Blackwell Max-Q Workstation Edition, 252 tok/s for $4.00/hr

Fewest GPUs: 2 x Rubin SXM, 1.1k tok/s

AMD 11 parts fit

Best value: 4 x Instinct MI300X 192GB, 395 tok/s for $9.56/hr

Fewest GPUs: 2 x Instinct MI455X, 1.2k tok/s

Huawei 8 parts fit

Fewest GPUs: 4 x Ascend 950DT, 197 tok/s

Intel 4 parts fit

Best value: 8 x Gaudi 2 96GB, 282 tok/s for $8.80/hr

Fewest GPUs: 8 x Gaudi 3 128GB, 497 tok/s

Google 2 parts fit

Fewest GPUs: 4 x TPU v7 192GB, 625 tok/s

Other 5 parts fit

Fewest GPUs: 8 x MTT S5000, 256 tok/s

Fleet what-if

Compare whole fleets at these settings: a thousand of one part against a hundred of another. Each fleet splits into replicas of its tensor-parallel width; GPUs left over sit idle.

A GPUs Part

B GPUs Part

Fleet letter	Fleet	Replicas	Idle	Tokens/s	Per hour	Power	Per M tokens
A	1,000 x H200 SXM 141GB	250 x TP4	0	81k tok/s	$2,480	700 kW	$8.55
B	100 x GB200 NVL72 GPU 186GB	25 x TP4	0	17k tok/s	$1,050	—	$17.23

Size Mistral Medium 3.5 128B for a tokens per second target

The calculator starts from your traffic instead of a fleet: set a target rate and read the replica count for every GPU.

Open in the LLM GPU Calculator

Frequently asked questions

How much GPU memory does Mistral Medium 3.5 128B need?

The weights take 128 GB at the native FP8 precision (255 GB at BF16, 128 GB at FP8, 71.8 GB at INT4). Each concurrent stream adds 352 KB of KV cache per token: 1.5 GB at 4,096 tokens and 11.8 GB at 32,768 tokens.

What is the bare minimum to run Mistral Medium 3.5 128B?

1 x Instinct MI355X OAM at FP8 with 4,096 tokens of context and one stream. The cheapest live rental for that footprint is 2 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $1.00 per hour.

How many H100s do you need to run Mistral Medium 3.5 128B?

2 H100s at FP8 for 4,096 tokens of context and one stream. For 32,768 tokens and 32 concurrent streams the tensor-parallel group is 8 H100s, and 1,000 tokens per second takes 16 H100s across 2 replicas.

How much does Mistral Medium 3.5 128B cost per million tokens on an H100?

About $7.81 per million output tokens at FP8, 32,768 tokens of context and 32 streams, using the cheapest live on-demand price of $1.83 per GPU-hour. The cheapest part per token is 8 x RTX PRO 6000 Blackwell Max-Q Workstation Edition at $4.42.

What precision does Mistral Medium 3.5 128B ship in?

The published checkpoint is FP8. FP8 halves the BF16 footprint with the accuracy the lab validated; INT4 is a further community quantisation.

All figures are roofline estimates from the model's config.json and each GPU's published memory, bandwidth and tensor-core peak, with fixed efficiency factors. Prices are the cheapest live on-demand listing per GPU when the site has one. They are a planning floor; a tuned serving stack can do better. Dense and multimodal; the total includes the vision encoder. Ships in FP8. The Modified MIT licence carries exceptions for very large companies.