d-Matrix Corsair Inference Server
Overview
Performance Metrics
| Precision | Peak (PFLOPS) | Per GPU (TFLOPS) | TFLOPS/W |
|---|---|---|---|
| INT8 | 19.2 | 2400.0 | — |
| INT4 | 76.8 | 9600.0 | — |
All figures are dense. Vendors commonly headline the sparse number, which is twice the dense one.
Configuration
- Accelerator
- Corsair
- Count
- 8
- Memory per GPU
- 256 GB
- Bandwidth per GPU
- —
Power & Efficiency
- Aggregate Power
- —
- TDP per GPU
- —
- INT8 Efficiency
- —
- Aggregate Bandwidth
- —
System Details
Interconnect
PCIe-based scale-up; 512 GB/s card-to-card DMX Bridge
Notes
Eight Corsair cards in one server, rated by d-Matrix at 19.2 PFLOPS of MXINT8 and 76.8 PFLOPS of MXINT4, both labelled dense on its own page. Memory is two-tier and unusual: 16 GB of on-die performance memory running at 1,200 TB/s, backed by up to 2 TB of capacity memory. There is no HBM anywhere in the machine. d-Matrix quotes 60,000 tokens per second at 1 ms per token for Llama3 8B on a single server. It publishes no power figure for the card, the server or the rack, so total system power is left blank rather than estimated. Note that the 1,200 TB/s belongs to the on-die SRAM and is deliberately not recorded as memory bandwidth, which on this site means DRAM bandwidth; d-Matrix publishes no bandwidth for the capacity tier.