August 2022 Summaries

In an increasingly virtual world, old business problems require new solutions, such as effectively documenting remote meetings via platforms like Zoom. Modern Automatic Speech Recognition technologies can help by automatically transcribing these remote meetings in real-time, allowing users to focus on the meeting without worrying about taking notes or missing important details. This tutorial guides you through implementing a solution to automatically transcribe a Zoom call by adding a recall.ai bot using AssemblyAI and Node.js, making it easier than ever to capture the contents of remote meetings while maintaining their flow and natural conversation.

In the data-driven world, sensitive information protection is vital for many companies to meet internal and external compliance requirements. With advances in Deep Learning research, AI models are becoming more powerful for tasks like PII Redaction. This article explores how these AI models are used to build Personally Identifiable Information (PII) redaction tools across industries such as call tracking, conversational intelligence, hiring software, and more. It presents five leading APIs for PII Redaction: AssemblyAI, Amazon Transcribe, Super.ai, Azure PII Redaction, and Private AI. Each API has unique features and pricing structures tailored to specific use cases and industries. The article also highlights the wide range of applications where PII Redaction tools can be integrated, including Customer Research Platforms, Hiring Intelligence Platforms, and Call Tracking Solutions.

This week's Deep Learning Paper Reviews discuss two research papers. The first paper applies continuous diffusion models to controllable natural language generation (NLG), improving text generation tasks through innovative "rounding" and "embedding" steps. Results show outperformance of existing methods such as PPLM and FUDGE, but a major bottleneck is the slow decoding speed. The second paper proposes representation pooling to sparsify transformer architectures, achieving sublinear time and memory complexity. An analysis shows a 1.8x speedup during training and 4.5x speedup during inference for long document summarization tasks, but might not be as useful for short input sequences.

Stable Diffusion is an AI text-to-image model developed by CompVis and released on August 22, 2021. It is a Latent Diffusion Model (LDM) that has demonstrated state-of-the-art performance in generating high-resolution images from natural language descriptions. The model uses a diffusion process to learn how to denoise images while also learning the underlying structure of images. This allows it to generate new images based on text inputs with remarkable fidelity and detail. It can generate images from a wide range of topics, including people, animals, objects, scenes, and more. Stable Diffusion is an open-source model that can be run locally or accessed via APIs. The official implementation uses Python and PyTorch for easy integration into existing workflows. To use Stable Diffusion, users need to install the necessary dependencies, including Python, CUDA (if using a GPU), and PyTorch. They then download the pre-trained weights of the model and place them in the correct directory structure. Once everything is set up, users can start generating images by providing a text prompt describing what they want to see. The model will then generate an image based on this description, which can be saved for later use or further processing. The quality of the generated images depends on several factors, including the complexity and specificity of the text prompt, the size and resolution of the output image, and the number of steps in the diffusion process. Experimenting with different prompts, settings, and parameter values can help users achieve better results.

MinImagen is a lightweight text-to-image model introduced by Google DeepMind in 2022. It demonstrates that it's possible to train a high quality text-to-image generator using a much smaller dataset and computational resources compared to models like DALL-E or Imagen. The MinImagen model consists of two main components: a base U-Net which generates low-resolution images, and a super-resolution U-Net that upscales the generated images to higher resolutions. The key innovation in MinImagen is using classifier-free guidance, where both the unguided (text-only) and guided (text + image caption) logits are used during training and sampling to improve the quality of generated images. Training a MinImagen model involves first training the base U-Net on low-resolution images paired with captions, followed by fine-tuning the super-resolution U-Net using the outputs from the base U-Net as inputs. The final MinImagen model can then be used to generate high quality images based on textual descriptions. In summary, MinImagen is a significant step forward in making advanced text-to-image models more accessible and computationally efficient, paving the way for further improvements and applications in this area.

"Barlow Twins" introduces a novel self-supervised learning (SSL) solution that doesn't require negative instances. Unlike most SSL algorithms based on contrastive learning, Barlow Twins avoids collapse by measuring the cross correlation matrix between outputs of two identical networks fed with distorted versions of a sample, aiming to make it as close to the identity matrix as possible. Additionally, batch normalization of features prior to the Barlow Twins loss is crucial for preventing collapse. This technique has shown competitive performance compared to state-of-the-art contrastive methods like SimCLR. In "Sparse MoEs Meet Efficient Ensembles," the paper explores using sparse Mixtures of Experts (MoE) and model ensembles together. MoE are neural networks that use dynamic routing at the token level to execute subgraphs, allowing for a larger parameter count than dense counterparts while maintaining the same compute requirements. The results show that sparse MoEs and static ensembles can have complementary features and benefits, providing higher accuracy, more robustness, and better calibration when used together. This suggests that even as the number of experts in an MoE increases, there is still additional value added by incorporating more models into a traditional model ensemble.

This tutorial will guide you through training and using a simplified version of Imagen called MinImagen. Imagen is an advanced text-to-image model developed by Google, which uses large language models to generate high-quality images based on given captions. MinIMagen is designed as a lightweight and accessible alternative to the original Imagen model, making it easier for developers to understand how such models work. This tutorial assumes that you have a basic understanding of Python and Pytorch. The code provided in this tutorial should be run in an environment with these dependencies installed. You can install them using pip: ```bash pip install torch transformers matplotlib skimage numpy ``` The full code for MinIMagen is available on GitHub, but the scripts have been simplified and modified for the purposes of this tutorial. To follow along, you can clone the repository or download the necessary files: ```bash git clone https://github.com/google-research/imagen.git cd imagen/minimagen # Download the Conceptual Captions dataset (for training) bash ./scripts/download_dataset.sh ``` Now, let's move on to setting up the environment and running the scripts! Setting Up The Environment To start, create a new Python virtual environment: ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` Now that your environment is set up, you can run the train script to generate "trained" MinIMagen instance weights: ```bash python minimagen_train.py ``` Generating Images with MinIMagen Once training is complete, you will see a new Training Directory, which stores all of the information from the training including model configurations and weights. To use this trained MinIMagen instance to generate images, run the inference script: ```bash python minimagen_inference.py ``` This should result in a new directory called generated_images_<TIMESTAMP>, which stores the captions used to generate the images, the Training Directory used to generate images, and the images themselves. The number in each image's filename corresponds to the index of the caption that was used to generate it. And that's it! You now have a basic understanding of how State-of-the-Art text-to-image models work and can use MinIMagen as a foundation for further exploration into this exciting field. For more Machine Learning content, feel free to check out more of our blog or YouTube channel. Alternatively, follow us on Twitter or follow our newsletter to stay in the loop for future content we drop. Follow the AssemblyAI Newsletter

This paper, titled "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis", presents a novel method for generating high-quality voice clones using just 5 seconds of audio from speakers not seen in the training set. Previously, state-of-the-art models needed tens of minutes of audio data. The authors achieved this by decoupling the speaker encoder and TTS network, reducing the data quality requirements for each step and enabling zero-shot learning. By training a large dataset on a self-supervised speaker verification task, the speaker encoder network generates fixed dimensional speaker embedding vectors that represent a speaker's voice characteristics independently from the audio content. These embeddings are then fed into a standard TTS pipeline with user input text embeddings to create log-mel spectrograms before being transformed into waveforms by a final vocoder network. This approach requires significantly less labeled data compared to previous end-to-end pipelines, which may open up opportunities for generating unlimited high-quality labeled data through various tweaks and modifications.

This week's Deep Learning Paper Recaps cover two important topics in Automatic Speech Recognition (ASR). The first paper, "Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition," presents a novel method to generate audio samples that are both imperceptible and robust. These adversarial samples can be played over the air without losing effectiveness. While the imperceptible attack has a high success rate, there is room for improvement in combining it with a robust attack. The second paper, "Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition," proposes using adapters to reduce the number of parameters needed when fine-tuning pretrained wav2vec models for each downstream task. This approach allows reusing 90% of the parameters for each task, reducing deployment costs and improving efficiency.