August 2019 Summaries

This tutorial provides a step-by-step guide on setting up a Horovod + Keras environment for multi-GPU training, requiring a machine with at least two GPUs and specific software installation. The necessary steps include installing NCCL2, Open MPI (optional), and Horovod in a Python3 virtual environment, followed by configuring the environment to run multi-GPU training jobs using either the `horovodrun` wrapper or the `mpirun` API. The tutorial concludes with a summary of the setup process, providing a one-stop installation script for all required steps after NCCL2 library download.

### Tracking system resource utilization during training with the Weights & Biases Dashboard` The Weights & Biases (wandb) Dashboard provides a centralized place to track system utilization during training, offering valuable insights into the training bottleneck and possible ways to uplift it. The dashboard displays 8 graphs, including CPU%, Memory%, Disk%, Network tx/rx bytes, GPU%, Temp (C), GPU Memory Access %, and GPU Memory Allocated %. These metrics help identify performance issues, such as high memory access or GPU utilization percentages, which can be optimized by adjusting training parameters, like batch size. Running experiments with real data also reveals changes in system utilization, highlighting the impact of data augmentation on CPU usage. The dashboard provides a comprehensive overview of system resource utilization during training, enabling users to optimize their machine learning experiments and improve overall performance.