October 2023 Summaries

Researchers from Carnegie Mellon University and Università Politecnica delle Marche propose a novel approach to combine Speaker Diarization (SD) and Automatic Speech Recognition (ASR) into a unified end-to-end framework. The objective is to simplify the speech processing pipeline while maintaining accurate speaker attribution and transcription. Traditional pipelines that couple SD and ASR rely on many distinct models, resulting in technical pitfalls like difficulty with hyperparameter tuning and model evaluation, computational overhead, and error propagation. SLIDAR, a 2-step approach to SD+ASR, involves analyzing fixed-length speech windows independently, employing a clustering mechanism for speaker identities, and maintaining linear computational costs relative to recording length. The proposed model demonstrates comparable performance to state-of-the-art methods despite using significantly less supervised training data.

Loop TV leverages Speech AI for brand protection by implementing advanced artificial intelligence models that analyze speech and detect unsuitable content in advertisements streamed on its channels. This helps businesses prevent inappropriate or competitive ads from appearing during their streaming services, ensuring consistent messaging and protecting their brand's integrity.

Bluedot is an AI-powered productivity app for asynchronous work that aims to automate documentation and reduce the number of meetings using AI. The company partnered with AssemblyAI to improve transcription accuracy, specifically in transcribing different accents and speaker recognition. After integrating Speech-to-Text and Audio Intelligence models from AssemblyAI, Bluedot achieved a significant increase in conversion rate from free to paid users, going from 2% to 3%. This was mainly due to the precise transcription provided by AssemblyAI, which helped detect technical terms, acronyms, and industry-specific language. The Custom Vocabulary feature of AssemblyAI enabled Bluedot to train its model based on customer-acquired acronyms, further enhancing accuracy. By leveraging AssemblyAI's solutions, Bluedot was able to reduce the workload of its engineers and achieve more with fewer resources.

AssemblyAI is preparing various product enhancements for Q4, including GDPR certification for data security compliance, improved punctuation and casing in transcriptions, reduced latency in real-time transcription, and enhanced LeMUR speaker labels. The company has also introduced new AssemblyAI Cookbooks to help developers easily integrate the platform into their products. Notable recent updates include a tutorial on building real-time closed-caption systems using React and an interview with Domenic Donato, VP of Technology at AssemblyAI, who was featured at Google Cloud Next. In addition, the blog has published tutorials on real-time transcription in Python, using audio data in LangChain with Python, and understanding large language models for Generative AI. The AssemblyAI YouTube channel features trending tutorials such as Retrieval Augmented Generation on audio data with LangChain and building a talking AI chatbot in five minutes.

The transcript appears to be a conversation between two people, discussing the latest tennis news and events, as well as mentioning their involvement in the Diversity and Inclusion committee for USDA. Speaker A congratulates Speaker B on her successful performance in an exhibition match. Speaker A also talks about how Speaker Diarization opens up significant analytical opportunities for companies by identifying each speaker and enabling product teams to analyze behaviors, identify patterns and trends, and inform business strategy. The transcript also mentions some challenges and limitations of Speaker Diarization models, such as the need for speakers to talk for more than 30 seconds, background noise affecting the model's ability to accurately assign speaker labels, and overtalk or interrupting conversations making it difficult for the model to appropriately assign speaker labels. The text also provides examples of how businesses are currently leveraging Speaker Diarization to create powerful transcription and analysis tools for their customers, such as virtual meeting and hiring intelligence platforms, conversation intelligence platforms, AI subtitle generators, and call centers. Finally, the text suggests some best practices for adding Speaker Diarization to enterprise applications, including keeping in mind that Speaker Diarization models work best when each speaker speaks for at least 30 uninterrupted seconds, and there is typically a limitation of the number of speakers a Speaker Diarization model can detect.

This weekly update includes news about the latest features and improvements of the multilingual speech-to-text tool, AssemblyAI. It now supports transcription across over 20 languages including Chinese, Hindi, Russian, Turkish, and Vietnamese. A new version (2.0) of the AssemblyAI Node SDK has been released, which is built from the ground up with TypeScript, offering a simplified interface for seamless interaction with the AssemblyAI API. The update also includes links to recent blog posts, interviews, and tutorials on using ASR models, integrating audio data into LlamaIndex, DALL-E 2 & 3, and creating AI audio apps in Python in just five minutes.

LlamaIndex is a flexible data framework that allows users to connect custom data sources to Large Language Models (LLMs). It enables easy storage and indexing of data, which can then be processed by LLMs. The integration of AssemblyAI into Llama Hub facilitates the transcription of audio files into text using just a few lines of code. This tutorial demonstrates how to transcribe audio files, load the transcribed text into LlamaIndex documents, and create a Query Engine to ask questions about spoken data.

The AssemblyAI Node SDK version 2.0 has been released, featuring a rewrite in TypeScript for easy API access using JavaScript or TypeScript. Users can transcribe audio/video files, perform LeMUR language model tasks on transcriptions, and transcribe real-time audio data using the new SDK. For more information, check out the GitHub repository README and the AssemblyAI documentation.

The speech and voice recognition market is projected to reach $60 billion by 2030, driven by advances in Artificial Intelligence (AI) research that have significantly improved the accuracy of speech recognition models. These developments have spurred demand for enterprise-facing Generative AI tools like DALLE-2, Stable Diffusion, and ChatGPT. Companies dealing with large amounts of customer data are exploring ways to use this new technology to build useful speech transcription tools and incorporate Generative AI features into their platforms. Automatic Speech Recognition (ASR) models employ AI to convert human speech into readable text asynchronously or synchronously, often with real-time transcription capabilities. ASR accuracy is measured by Word Error Rate (WER), which takes into account substitutions, deletions, and insertions in a transcription text compared to a human transcription. WER calculations may vary based on factors like capitalization, punctuation, spelling, and dataset relevance. Ensuring ASR accuracy matters, especially for product teams building Generative AI tools and features on top of transcription data. Accurate transcription is crucial in ensuring the effectiveness and reliability of these intelligent tools and features. Real-world use cases demonstrate how ASR can significantly augment productivity, reduce manual tasks, and provide valuable insights across various industries. Integrating highly accurate ASR models with AI-powered tools and working with an AI partner can help companies build precise, high-performing applications faster.

In recent product updates, AssemblyAI introduced a new usage dashboard with detailed analytics displaying model-specific usage over time. They have also integrated support for the AWS marketplace into their web application's dashboard/account management pages. The team resolved an issue where LeMUR occasionally faced challenges in handling extremely short transcripts. Additionally, AssemblyAI LlamaIndex Integration is now available, allowing users to transcribe audio files and load them onto documents easily. Furthermore, they announced the winners of the AssemblyAI-Streamlit Hackathon, and Dylan Fox will be appearing on Accel's podcast series.

In this tutorial, we learned how to use the AssemblyAI Python SDK to perform real-time transcription of audio streams using a WebSocket connection. We saw how easy it is to transcribe speech in real time and display the text on screen with just a few lines of code. Firstly, we installed the necessary dependencies for working with the Python SDK, including the `websockets` and `aai` libraries. Then, we defined two handler functions that would be called when receiving data from the WebSocket or an error occurred. In these handlers, we determined if the transcript was a final transcript (indicating the end of an utterance) or not, and printed it to the console accordingly. Next, we created another function for handling partial transcripts, which are sent while still processing audio data. These partial transcripts contain all previous words in the utterance, so by printing only the new text at the end of each partial transcript, we could make it appear as if only the delta (i.e., the new words) were being displayed since the last message. We then wrote an error handler that simply printed any errors that occurred during transcription. Finally, we created our main script code by instantiating a RealtimeTranscriber object and passing in our two handlers as well as the sample rate for the audio stream. We opened a WebSocket connection using this object's connect method, then opened a microphone stream to pass the audio data into the transcriber's stream method. Once done transcribing, we closed the WebSocket connection to clean up all loose ends before exiting the script. In addition, we saw how to define open and closing handlers for when the WebSocket is opened or closed. When these were added to our transcriber definition, they printed a message indicating when each event occurred. This tutorial provided an overview of real-time transcription using Python with the AssemblyAI SDK. By following along, you should now have a good understanding of how this process works and be able to implement similar functionality in your own projects.

Sales intelligence platforms are becoming more advanced with the integration of Speech AI technology, which helps analyze and extract valuable insights from audio data in sales calls, demos, team meetings, and customer service conversations. This leads to personalized outreach strategies, predictive analytics for prioritizing leads, sentiment analysis, and voice recognition capabilities. AssemblyAI's Speech AI system can improve sales intelligence platforms by analyzing hours of audio data, providing quick summaries, sentiment analysis, real-time transcriptions, automatic identification of action items from calls, and feedback requests for areas of improvement. Companies like Jiminny have already successfully implemented Speech AI technology to secure higher win rates for their customers.

AssemblyAI introduces a brand new Usage Dashboard, allowing users to view their usage and spend data with more flexibility through new date range options. The dashboard has also been redesigned for greater user-friendliness. Community Spotlight highlights various projects created by users using AssemblyAI's API, such as video content repurposing tools and fitness apps that generate exercise routines from YouTube videos. Blog posts cover topics like retrieval augmented generation on audio data, speech AI systems in podcasting, understanding DALL-E 2 and the newly announced DALL-E 3, and eight ways ASR can increase business efficiency. AssemblyAI encourages users to share their innovations and offers resources for building applications using spoken data.