Getting the GPU utilization of NVIDIA playing cards with the Linux dstat instrument

The dstat is an superior little instrument which lets you get useful resource statistics on your Linux field. It has a modular structure which lets you develop extra plugins and it’s simple to make use of. Lately I used to be profiling a Deep Studying pipeline developed with Keras and Tensorflow and I wanted detailed statistics in regards to the CPU, Laborious Disk and GPU utilization. The primary two can be found out-of-the-box by dstat, nonetheless so far as I do know there isn’t any plugin for monitoring GPU utilization for NVIDIA graphics playing cards.

Fortunately it’s tremendous simple to put in writing a python plugin for dstat. I’ve already despatched a pull-request on the official repo however since new variations are launched comparatively hardly ever listed below are some directions on the best way to arrange the dstat NVIDIA GPU utilization plugin in your field.

Set up

The next instructions are examined on Ubuntu 16.04 and they’re going to enable you to set up dstat, the Python NVIDIA Administration Library and my dstat nvidia plugin:

sudo apt-get set up dstat #set up dstat
sudo pip set up nvidia-ml-py #set up Python NVIDIA Administration Library
wget https://uncooked.githubusercontent.com/datumbox/dstat/grasp/plugins/dstat_nvidia_gpu.py
sudo mv dstat_nvidia_gpu.py /usr/share/dstat/ #transfer file to the plugins listing of dstat

To get all of the default statistics together with GPU utilization (proportion) sort the next command:

dstat -a --nvidia-gpu

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- gpu-u
usr sys idl wai hiq siq| learn  writ| recv  ship|  in   out | int   csw |whole
  2   1  96   0   0   0|5816k   15M|   0     0 |   0     0 |  45k   98k|   68
  0   1  98   0   0   0|  57M  128k| 104B  902B|   0     0 |  42k   85k|   50
  8   7  84   1   0   0| 152M    0 | 292B  448B|   0     0 |  52k   93k|   39
  1   1  97   1   0   0| 111M    0 |  52B  374B|   0     0 |  51k  116k|   62
  0   1  98   1   0   0| 129M    0 |  80B  416B|   0     0 |  43k   85k|   92
  0   2  98   0   0   0|   0     0 |  52B  374B|   0     0 |  41k   83k|   81

To get all of the utilization statistics for every GPU use the next command:

dstat --nvidia-gpu -f
-------------------------------------------gpu-usage-nvidia------------------------------------------
whole  gpu0  gpu1  gpu2  gpu3  gpu4  gpu5  gpu6  gpu7  gpu8  gpu9 gpu10 gpu11 gpu12 gpu13 gpu14 gpu15
   19    23    22    21    21    20    22    23    25    15    18    16    16    16    18    16    14
   18    21    20    18    22    21    21    22    21    15    15    14    14    14    15    16    13
   10    14     9    13     8     9    11     9    12     9     9    10    10     8     7     9     9
   18    20    22    19    21    20    21    21    22    14    15    14    15    14    15    15    15
   20    24    22    23    24    25    22    22    22    16    16    16    16    16    16    18    16
   15    21    18    19    18    17    17    16    18    14    13    13    14    13    12    11    11
   20    24    22    22    24    25    23    24    22    16    18    16    14    17    17    17    15
   19    29    18    23    21    22    21    20    21    18    16    16    18    14    14    17    17

The way it works

The plugin fetches the variety of out there GPUs on the system and samples 10 instances the utilization metric for every GPU. Sampling a number of instances will hopefully return smoother metrics than getting a single measurement. After that it averages the utilization throughout all GPUs and returns the outcomes to the person. The supply code of the plugin is obtainable right here.

Hope you get pleasure from it, joyful GPU programming! 🙂