February 2019 Summaries

Performing stress testing on Linux systems involves using multiple tools, including `stress`, `gpu_burn`, and monitoring tools like `htop`, `iotop`, and `nvidia-smi`. These tools are used to simulate various loads on the system's CPU, GPU, and I/O, allowing users to test their system's performance and identify potential bottlenecks. The stress tests can be customized with parameters such as number of workers, memory allocation, and disk usage, and can run for a specified duration. By running these tests, users can ensure that their system is working efficiently and meeting the required specifications.

OpenAI's GPT-2 text generator is an unsupervised language model that achieves state-of-the-art performance on many language modeling benchmarks and performs rudimentary reading comprehension, machine translation, question answering, and summarization. To run the code yourself, you need to install system-wide dependencies such as CUDA, cuDNN, and NVIDIA graphics drivers, then clone the GPT-2 repository, create a virtual environment, install Python dependencies and the GPT-2 code, and set k=40 in interactive_conditional_samples.py. You can then run the model using python3 src/interactive_conditional_samples.py. The larger 345M model has been released, which produces more accurate sounding code tutorials.

The text compares the cost of buying vs. renting a cloud GPU server, specifically focusing on Deep Learning workloads. A server with similar hardware to AWS's p3dn.24xlarge is selected for comparison, which is a Tesla V100 Server from Lambda Hyperplane. The study finds that the purchased Tesla V100 Server is 2.6% faster than the AWS p3dn.24xlarge for FP32 training and 3.2% faster for FP16 training. Additionally, the TCO (Total Cost of Ownership) analysis shows that the Hyperplane on-prem server has a lower total cost compared to the p3dn.24xlarge over a 3-year period, with savings ranging from $69,441 to $184,008. The study highlights the benefits of managing software and hardware in-house, as well as the reduced costs associated with purchasing a server upfront versus renting it on AWS. The results also suggest that while cloud services offer ease of use for real-time applications, Deep Learning workloads may not benefit from these advantages.

To build a GPU accelerated Docker container with TensorFlow or PyTorch in it, you can follow the steps outlined in this tutorial. First, install Lambda Stack on your Ubuntu host system, followed by installing Docker and nvidia-container-toolkit. Next, build an open source Lambda Stack Dockerfile to create a custom image, ensuring its creation by running `sudo docker image list`. You then run a test job within the container using the `docker run` command. Finally, upload your container image to a registry, such as Docker Hub, and verify it works on any new computer after installation. With these steps, you can now use Lambda Stack + Lambda Stack Dockerfiles for GPU accelerated deep learning containers.

You can install Ubuntu 18.04 and then Lambda Stack, a tool that allows developers to deploy and manage their applications on Amazon Web Services (AWS). This installation process is straightforward and requires only a few commands. The Lambda Stack installation includes support for NVIDIA Titan RTX and 2080 Ti (Turing) GPU accelerators, which enables developers to utilize the performance of these high-end graphics cards.

This tutorial provides a step-by-step guide on creating a character-based text generator using a simple two-layer LSTM. The data preparation process involves reading the training data, processing it into numerical representations, and preparing examples for training. The network architecture is designed to learn from these examples and generate new characters based on the input sentence. The model uses cross-entropy loss during training and softmax function during testing. The generated text can be controlled by adjusting the temperature hyperparameter. The tutorial includes code snippets and a demo repository for further exploration.

BERT, a pre-training language representation developed by Google, has achieved state-of-the-art results on various Natural Language Processing tasks. However, its official TPU-friendly implementation only supports single GPU usage at present. This blog post aims to make BERT work with multiple GPUs using Horovod, a framework for parallelizing tasks. The authors have made several changes to the original BERT implementation, including importing Horovod's Tensorflow backend, initializing the library, pinning each worker to a GPU, and adapting gradient clipping accordingly. By leveraging these modifications, the authors demonstrate improved performance on multiple GPUs, with throughput increases ranging from 126.92 examples/sec for sentence classification (2 GPUs) to 231.26 examples/sec for the same task (4 GPUs). The authors also highlight potential pitfalls, such as using unsynchronized models across different workers and adapting gradient clipping accordingly. By following these changes and modifications, developers can adapt BERT for multi-GPU usage, leading to improved performance on various NLP tasks.