2

If I'm running a Keras model on some Nvidia GPUs which are connected via NVLink, how can I monitor the usage of the NVLink connections?

I want to be sure the NVLink connections are being used and see at what rate.

user3731622
  • 121
  • 1
  • 3

1 Answers1

1

nvidia-smi nvlink

This command shows various information about nvlink including usage. If nvlink connections are utilized, usage should go up during training.

nvidia-smi nvlink -h

Shows available performance counters on present cards.

"NVLink Usage Counters" section in this tutorial shows how to see if data is being transferred across nvlink.

https://www.exxactcorp.com/blog/HPC/exploring-nvidia-nvlink-nvidia-smi-commands

Shamit Verma
  • 2,239
  • 1
  • 8
  • 14
  • 1
    Hi @Shamit Verma, the link has obsoleted, did u mean this link: https://www.exxactcorp.com/blog/HPC/exploring-nvidia-nvlink-nvidia-smi-commands . using `nvidia-smi nvlink -s` returns for ```GPU 0 Link 0: 14.062 GB/s Link 1: 14.062 GB/s Link 2: 14.062 GB/s Link 3: 14.062 GB/s``` same for GPU 1. does it mean the nvlink is working? I did not find any specific command about nvlink working status. Could u help? – bim Apr 12 '23 at 16:12
  • 1
    Thanks, updated the link – Shamit Verma Apr 15 '23 at 05:57