January 2019 Summaries

The presentation discusses the challenges of training deep learning models on-premises with GPU infrastructure, highlighting the need for scalable and efficient solutions. The author proposes a framework called Lambda, which enables the creation of custom, on-prem GPU training infrastructures tailored to specific deep learning use cases. This approach allows developers to build and deploy their own GPU-accelerated environments, reducing reliance on cloud-based services and enabling more control over data privacy and security. By leveraging Lambda, organizations can optimize their deep learning workflows for better performance, scalability, and cost-effectiveness.

The text discusses the reproduction of the current state-of-the-art ImageNet training performance on a single Turing GPU server, achieving 93% Top-5 accuracy in just 2.36 hours. This was made possible by using dynamic-size images and replacing fully connected layers with global pooling layers, which reduced unnecessary preprocessing and allowed for more efficient inference. Additionally, the team employed progressive training with images of multiple resolutions, increasing the resolution step-by-step while adjusting the batch size and learning rate to achieve optimal performance. The results demonstrate a significant reduction in training time compared to previous approaches, showcasing the effectiveness of these techniques in achieving state-of-the-art performance on ImageNet.

The ResNet9 model has achieved a significant speedup in training on CIFAR10, with an accuracy of 94% in just 75 seconds using a single V100 GPU. This is a substantial improvement over the previous winning entry from FastAI, which required 8x more GPUs and took nearly twice as long to train. The model's performance was achieved through a series of modifications, including removing unnecessary layers, optimizing batch size and random number generation, and using single-precision for batch norm. The most significant improvement came from optimizing the residual network architecture, with an estimated 27% reduction in training time. Despite this impressive achievement, there is still room for further improvement, with potential speedups of up to 2X possible if compute efficiency can be realized.

The task of object detection is to identify "what" objects are inside an image and "where" they are. Object detection has been a central problem in computer vision and pattern recognition, inheriting challenges from image classification such as robustness to noise, transformations, occlusions, and introducing new challenges like detecting multiple instances and identifying their precise locations. The Single Shot Detector (SSD) is a multi-scale sliding window detector that leverages deep CNNs for both classification and localization tasks. SSD makes detection more robust by leveraging deep features and allows feature sharing between the classification task and the localization task. The network outputs a prediction map with class confidence and bounding box information, which is then processed using priorbox to select ground truth objects and compute loss. Priorbox uses a simple distance-based heuristic to create ground truth predictions, including backgrounds where no matched object can be found. SSD uses hard negative mining to address the problem of imbalance between foreground and background samples. The network also employs data augmentation strategies like "zoom in" and "zoom out" to improve performance on detecting large and small objects, respectively. Pre-trained feature extractors and L2 normalization are used, with modifications made to the VGG_16 model. Post-processing involves filtering out weak detections using confidence thresholds and performing non-maximum suppression to curate results.