September 2018 Summaries

Hyper-parameter tuning is a crucial step in improving the performance of neural networks, particularly in image classification. It involves systematically searching for optimal hyper-parameters such as learning rate, optimizer, and batch size to achieve better results. The search can be done using various strategies including coarse to fine search, random search, grid search, and Bayesian optimization. In this tutorial, a demo suite is provided to showcase how to find the optimal optimizer and learning rate using the tuner function in TensorFlow. A good practice for hyper-parameter search includes sampling hyper-parameters in scale space, using random search instead of grid search, searching from coarse to fine, and considering Bayesian optimization. The demo provides a YAML configuration file that defines the number of trials, fixed parameters, and hyper-parameters to be searched, allowing users to tune the optimal hyper-parameters for ResNet32 on CIFAR10.

This tutorial discusses transfer learning using TensorFlow for image classification. It highlights the importance of minimizing data collection and labeling efforts by leveraging pre-trained models trained on large datasets like ImageNet. The tutorial covers two main approaches to transfer learning: 1) freezing all but the last layer of a pre-trained model (feature extractor approach), which is useful when the new dataset is closely related to the old one, and 2) allowing all layers to remain trainable (fine-tune-all-layers approach), which is better when the new dataset has more data than the old one. The tutorial demonstrates how to use transfer learning with ResNet50, InceptionV4, and NasNet-A-Large models on the Stanford Dogs dataset. It also provides implementation details, including how to initialize weights from a pre-trained model, train the network with new data, and handle batch normalization layers. The key takeaway is that transfer learning can significantly reduce the burden of data collection and labeling by adapting pre-trained models to new tasks.

A TensorFlow demo that guides users through building an image classification application using the ResNet32 network and the CIFAR10 dataset. The demo consists of three main components: Inputter, Modeler, and Runner. The Inputter encapsulates the data pipeline into a function `input_fn`, which reads data from disk, shuffles, preprocesses, creates batches, and prefetches. The Modeler encapsulates the forward pass and computation of loss, gradient, and accuracy in the `model_fn` method. The Runner orchestrates the execution of the Inputter and Modeler, distributes the workload across multiple hardware devices, and uses callbacks to perform auxiliary tasks such as logging statistics and saving trained models. The demo includes an example code snippet that demonstrates how to create these components and run the application.