How to Set Up NVIDIA DCGM for GPU Metrics on Ubuntu

NVIDIA Data Center GPU Manager, DCGM, is the monitoring layer under almost every GPU dashboard you have seen: the Prometheus exporter in Kubernetes, the GPU panels in Grafana, the health checks a cloud runs before it hands you a node. It is free, it ships in NVIDIA’s own package repository, and on a single Ubuntu machine it takes about ten minutes to go from nothing to live metrics.

This guide installs DCGM 4 on Ubuntu 22.04 or 24.04, starts the host engine as a service, reads the first metrics on the command line, and then exports them to Prometheus. Every command, package name and field number was checked against NVIDIA’s current DCGM documentation and the live package repository on the date at the top of the page; the repository served DCGM 4.7 and exporter 4.8 at the time.

What DCGM gives you that nvidia-smi does not

nvidia-smi reports GPU utilization, which is the share of the last sample period during which at least one kernel was running. A GPU can show 100% there while most of its silicon sits idle.

DCGM adds the profiling metrics: the share of time the SMs were active, how full they were, how busy the tensor pipe was, how hard the memory bus was working, and the bytes moving over PCIe and NVLink. Those are the numbers that tell you whether a training job is compute-bound, memory-bound or waiting on the host.

It also adds the operational layer: ECC and XID error counters, thermal and power events, NVLink health, background health watches you can query before a job, and an active diagnostic suite that stress-tests a GPU you suspect is failing. All of it is readable from a command line, a Python binding, or a Prometheus endpoint.

Before you start

You need:

• Ubuntu 22.04 LTS or 24.04 LTS, on x86_64 or arm64. NVIDIA lists 26.04 LTS as supported too.
• An NVIDIA GPU: any datacenter part from Kepler (K80) onwards, or a GeForce or workstation card from Maxwell onwards. The profiling metrics this guide leans on need a datacenter GPU from Volta onwards, and the deeper diagnostics need a datacenter GPU.
• The NVIDIA driver already installed, version R450 or newer. If nvidia-smi prints a table with your GPU in it, you are ready. If it does not, install the driver first; the same NVIDIA repository you add in step 1 carries it.
• sudo on the machine. The host engine runs as root so it can read the profiling counters.

Note the CUDA version your driver reports. You will use its major number in step 2.

nvidia-smi -q | grep -E 'Driver Version|CUDA Version'

Step 1: add NVIDIA’s package repository

DCGM is published in the CUDA network repository, not in Ubuntu’s own archives. Adding the repository means installing NVIDIA’s signing keyring package, then refreshing apt.

Ubuntu 24.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update

Ubuntu 22.04 is the same with ubuntu2204 in the URL. On an arm64 server such as a Grace system, replace x86_64 with sbsa; that is the repository that carries DCGM, whatever NVIDIA’s architecture table suggests.

If an older DCGM 3 is installed, stop its service and remove it before going further. DCGM 4 uses new package names and declares a conflict with the old one:

sudo systemctl list-unit-files nvidia-dcgm.service > /dev/null && sudo systemctl stop nvidia-dcgm
sudo dpkg --list datacenter-gpu-manager &> /dev/null && sudo apt purge --yes datacenter-gpu-manager
sudo dpkg --list datacenter-gpu-manager-config &> /dev/null && sudo apt purge --yes datacenter-gpu-manager-config

Step 2: install DCGM

DCGM 4 ships one package per CUDA major version. Set the variable to the CUDA major version your driver reports (12 for a 12.x driver, 13 for 13.x) and install. One exception: on a Maxwell, Pascal or Volta GPU with a 580-series driver, use 12 even though the driver reports 13.0, because CUDA 13 dropped those architectures.

CUDA_VERSION=12
sudo apt-get install --yes --install-recommends datacenter-gpu-manager-4-cuda${CUDA_VERSION}

That pulls in the CUDA-independent core package, the dcgmi command-line tool, the nv-hostengine daemon with its systemd unit, the diagnostic plugins built for your CUDA version, and the Python bindings, which land under /usr/share/datacenter-gpu-manager-4/bindings/python3/. The --install-recommends flag adds NVIDIA’s proprietary components; leave it out with --no-install-recommends if you only want the open-source parts.

Two optional packages:

# Headers and libraries for writing your own DCGM client
sudo apt-get install --yes datacenter-gpu-manager-4-dev

# Multi-node diagnostics (CUDA 12 or later)
sudo apt-get install --yes datacenter-gpu-manager-4-multinode-cuda${CUDA_VERSION}

Step 3: start the host engine

DCGM is a client-server system. The server is nv-hostengine; it talks to the driver, keeps the metric watches running, and answers clients on 127.0.0.1 port 5555. On Ubuntu it is managed as the nvidia-dcgm systemd service.

sudo systemctl enable --now nvidia-dcgm
systemctl status nvidia-dcgm --no-pager

You should see active (running). Then ask the engine what it found:

dcgmi discovery -l

A healthy result lists every GPU with its bus ID and UUID, and any NVSwitches on an HGX or DGX board. If you get Error: unable to establish a connection to the specified host, the service is not running: journalctl -u nvidia-dcgm and /var/log/nv-hostengine.log say why.

Step 4: read metrics with dcgmi

dcgmi dmon streams metrics to the terminal, one row per GPU per sample. You choose fields by numeric ID. Start with the profiling trio that answers “what is this GPU actually doing”:

dcgmi dmon -e 1001,1004,1005 -d 1000

That samples graphics engine activity, tensor pipe activity and memory bandwidth utilization once a second (-d is the interval in milliseconds; 1000 is the default, and 100 is the floor for profiling fields). Press Ctrl+C to stop. Add the fields you care about to the list; these are the ones worth knowing by number:

The full list is in NVIDIA’s field identifier reference. dcgmi dmon -l prints every field ID the installed dcgmi knows about, without checking what your GPU supports; for the profiling fields, dcgmi profile -l -i 0 does check, and also shows which of them GPU 0 can sample together.

To check the setup end to end without a real workload, DCGM ships a load generator. This puts a tensor-core load on GPU 0 for 30 seconds. With --no-dcgm-validation it only generates the load, so you watch the result with dcgmi in a second terminal:

# Terminal 1: load GPU 0 (omit -i and it loads every GPU in the machine)
/usr/bin/dcgmproftester12 --no-dcgm-validation -i 0 -t 1004 -d 30

# Terminal 2: watch tensor activity on GPU 0
dcgmi dmon -e 1004 -i gpu:0

You should see tensor activity climb towards 1.0. Use dcgmproftester13 on a CUDA 13 install, except on Maxwell, Pascal and Volta, which stay on dcgmproftester12.

Step 5: export to Prometheus

For anything beyond a terminal you want dcgm-exporter, which turns DCGM fields into a Prometheus endpoint on port 9400. There are two ways to run it.

As a package. NVIDIA publishes the exporter in the same repository:

sudo apt-get install --yes datacenter-gpu-manager-exporter
sudo systemctl enable --now nvidia-dcgm-exporter
curl --fail http://localhost:9400/metrics | head -20

The response is Prometheus text with metric names starting DCGM_. Two things to know about the packaged service:

• By default it runs its own embedded host engine, a second one beside the nvidia-dcgm service from step 3, and both will want the profiling counters. To make it a client of the engine you already started, add -r localhost:5555 to its command line.
• The set of metrics is controlled by /etc/dcgm-exporter/default-counters.csv, a three-column file of DCGM field name (not number), Prometheus type and help text. The default file already carries fields 100, 150, 155, 203, 252, 230, 1001, 1004, 1005, 1009 and 1010 from the table above. DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY are there but commented out; uncomment them. The NVLink pair is absent; add two lines for DCGM_FI_PROF_NVLINK_TX_BYTES and DCGM_FI_PROF_NVLINK_RX_BYTES. The exporter samples every 30 seconds by default (--collect-interval, in milliseconds).

Both changes go in a systemd drop-in rather than the vendor unit. sudo systemctl edit nvidia-dcgm-exporter.service opens one; an empty ExecStart= line clears the packaged command before you set yours:

[Service]
ExecStart=
ExecStart=/usr/bin/dcgm-exporter -r localhost:5555 -f /etc/dcgm-exporter/custom-counters.csv

As a container. If you would rather not install anything but Docker and the NVIDIA Container Toolkit:

docker run -d --rm --name dcgm-exporter 
  --gpus all --cap-add SYS_ADMIN 
  -p 9400:9400 
  nvcr.io/nvidia/k8s/dcgm-exporter:latest
curl --fail http://localhost:9400/metrics | head -20

The container also runs its own embedded host engine. --cap-add SYS_ADMIN is what lets it read the profiling counters; drop it if you do not need them, remembering that five of the default CSV’s fields are profiling fields and will go missing. To reuse the host engine from step 3 instead, run the container with --network host in place of -p 9400:9400 (the engine listens only on the host’s loopback, which a bridged container cannot reach) and add -r localhost:5555 after the image name. Pin a versioned tag from NVIDIA’s container catalog in anything that will run for longer than an afternoon; the tags pair a DCGM version with an exporter version, such as 4.6.1-4.8.4-distroless, and the exporter’s DCGM version must be at least the host engine’s.

Then add a scrape job to Prometheus:

scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets: ['gpu-host-1:9400']

Each series carries gpu, UUID, pci_bus_id, device, modelName and hostname labels, so one dashboard covers every node.

Step 6: health checks and diagnostics

DCGM works on groups of GPUs. The special group g means every GPU in the system, which is enough for a single host. Turn on health watches for it, give them a minute to collect samples, then query them:

dcgmi health -g g -s a
dcgmi health -g g -c

The -s flags choose what to watch: p PCIe, m memory, i InfoROM, t thermal and power, n NVLink, d driver, x ConnectX, e IMEX, or a for all, used on its own. The check returns Healthy, Warning or Failure with the reason. The watches are passive, so the check is cheap enough to run before every job, but they do not survive a host engine restart: set them once at boot, not just before you ask.

When a GPU looks wrong, run the diagnostic. Level 1 takes seconds and checks the software stack and basic health. Level 2 adds memory and PCIe tests and takes about two minutes. Level 3 adds the stress, memory bandwidth and targeted power tests, about fifteen minutes, and needs the GPU otherwise idle. A level 4 with extended memory and pulse tests exists for when a GPU still passes 3 and you still do not trust it.

dcgmi diag -r 1
dcgmi diag -r 2

Levels 2 and 3 are the tests that find the memory errors and thermal problems a cloud provider would pull a node for. On GeForce and workstation cards only level 1 is available.

Troubleshooting

• “unable to establish a connection to the specified host” from any dcgmi command: the host engine is not running. sudo systemctl start nvidia-dcgm, then read journalctl -u nvidia-dcgm or /var/log/nv-hostengine.log.
• Profiling fields show N/A or Not Supported: the GPU is not a Volta-or-newer datacenter part, the driver is older than DCGM expects, DCGM profiling is paused, or another process owns the counters. Check nvidia-smi for the driver, run dcgmi profile --resume, and close Nsight.
• apt cannot find datacenter-gpu-manager-4-cuda12: the repository was not added for your Ubuntu release, or apt-get update was not run after the keyring install. apt-cache policy datacenter-gpu-manager-4-cuda12 shows what apt can see.
• The exporter container starts and exits: Docker cannot see the GPUs. Install the NVIDIA Container Toolkit and confirm docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi works first.
• Two host engines fighting: the nvidia-dcgm service, the packaged exporter and a container exporter each start their own engine by default, and only one can hold the profiling counters. Run one engine and point everything else at it with -r.

Reading the numbers against the spec sheet

A tensor activity of 0.6 means the tensor pipe was busy for 60% of cycles, averaged across the SMs: NVIDIA notes that 0.2 can be a fifth of the SMs at full tilt or all of them at a fifth. It does not mean you are getting 60% of the datasheet TFLOPS either: that depends on precision, on clocks, and on whether the figure you are comparing with is the dense or the sparse one. The dense vs sparse guide covers the second trap. For the first, every GPU page on this site lists peak throughput by precision, so you can put the DCGM ratio next to the number it is a ratio of.