December 2023 Summaries

In 2023, AssemblyAI launched several products and features for customers including Conformer-1 and Conformer-2 AI models for speech recognition, LeMUR framework for applying LLMs to spoken data, a partnership with AWS Marketplace, enhanced AI models for PII Redaction, Entity Detection, and punctuation and casing, a redesigned no-code playground, Series C funding round of $50 million, LeMUR Cookbooks for building applications using LLMs, and various top blog posts.

Speech AI can be integrated into Learning Management Systems (LMS) and educational applications to improve functionality, user experience, and accessibility. Here are five ways to enhance an LMS with speech recognition technology:1. Make course content more accessible2. Enhance searchability within the platform3. Help educators evaluate learners’ reading comprehension4. Build a feedback loop between platforms, educators, and learners5. Equip learners with smart study toolsIntegrating Speech AI into an LMS can make education more equitable, efficient, and effective for both educators and students alike.

This guide teaches how to create VTT (.vtt) files for videos using Python. A VTT file is a text format containing supplementary information such as subtitles or captions, commonly used for video subtitles. The syntax closely resembles that of SRT files but with some differences. To generate VTT files in Python, the AssemblyAI Python SDK and an AssemblyAI API key are required. The code demonstrates how to transcribe a YouTube video and convert the transcription into a VTT file. The captions can be limited to a specified maximum number of characters per caption by using the chars_per_caption parameter.

AssemblyAI has introduced a new punctuation and casing model for real-time transcription, offering improvements in question marks attribution, handling of challenging linguistic types such as mixed-case words, acronyms, and capital case, as well as an overall 17% relative improvement on average across test datasets for predicting upper-case letter classification. The company has also released version 4 of its JavaScript SDK, which now supports various runtime environments including web browsers, Bun, Cloudflare Workers, and more. Additionally, they have shared insights into recent advancements in universal audio understanding, the creation of SRT files for videos in Python, key phrase detection in audio files using Python, speech-to-text conversions in Java, building no-code AI app prototypes with Rivet, and running LLMs locally. Reference(s): https://github.com/assemblyai/assemblyai/blob/main/blog/new_real-time_model.md

This guide demonstrates how to create SRT files for videos using Python and the AssemblyAI Python SDK. An SRT file is a subtitle file format that includes timing information and text for each subtitle. The guide outlines the structure of an SRT file, provides examples, and explains how to install the necessary dependencies and set up a project in Python. It also shows how to create SRT files using YouTube video URLs and modify the maximum number of characters per caption.

AssemblyAI has expanded its documentation to include tutorials for using their Speech-to-Text API to build voice applications, SDKs for Java and Go languages, Webhooks docs, and integrations with tools like LangChain, LlamaIndex, Zapier. They have also raised $50M in Series C funding led by Accel, allowing them to develop superior Speech AI models. Additionally, they provide a blog with insights on how to integrate LLMs into speech processing tasks and explanations of various AI concepts.

Alibaba Group researchers have introduced Qwen-Audio, a large-scale audio-language model that significantly enhances AI systems' ability to process and reason about various audio signals. Unlike previous models, Qwen-Audio integrates a pre-training learning objective spanning over 30 distinct tasks and accommodating multiple languages, setting a new standard in universal audio understanding capabilities. The model demonstrates unparalleled performance across an extensive array of audio datasets, bringing the potential for more sophisticated audio understanding capabilities that align with advancements seen in other AI domains. Qwen-Audio's capabilities include multilingual ASR and translation, multiple audio analysis, sound understanding and reasoning, audio-motivated creative writing, music appreciation, and speech editing with tool usage.

LlamaIndex is a data framework that enables connecting custom data sources to Large Language Models (LLMs). It allows for storage, indexing, and use of data with LLMs in application development. However, since LLMs work only with textual data, audio files need to be transcribed first using tools like AssemblyAI's speech-to-text feature. There is an integration for LlamaIndex Python through LlamaHub which makes it easier to start using the feature. A sample LlamaIndex.TS application can be found that answers questions about an audio file, transcribing the file and using OpenAI for responses. Check out more details in AssemblyAI's LlamaIndex.TS integration docs or learn how to build a Q&A application for audio files. Additionally, AssemblyAI offers LeMUR (Leveraging Large Language Models to Understand Recognized Speech), which can perform tasks over large amounts of long audio files using an LLM and its API.

In this tutorial, you will learn how to create an application that can answer questions about an audio file using LlamaIndex.TS and its new integration with AssemblyAI's speech-to-text APIs. LlamaIndex is a flexible data framework for connecting custom data sources to large language models (LLMs). However, LLMs only operate on textual data and do not understand what is said in audio files. With the recent contribution of an integration with AssemblyAI to LlamaIndex.TS, developers can now transcribe audio files to text within LlamaIndex. The tutorial guides you through setting up a TypeScript Node.js project, configuring environment variables for OpenAI and AssemblyAI API keys, using LlamaIndex.TS to build a Q&A application, transcribing an audio file with the AudioTranscriptReader provided by AssemblyAI's integration, and querying the index for answers. Finally, you can explore LeMUR in the Playground using the link provided.

AssemblyAI has raised $50M in Series C funding led by Accel, bringing its total funds raised to $115M. The company specializes in creating superhuman Speech AI models that unlock new opportunities for organizations across various industries. Over the past two years, advancements in larger datasets, better compute, and new neural network architectures have enabled significant progress in AI models. AssemblyAI's Conformer-2 model demonstrates industry-leading accuracy and robustness for speech-to-text and speaker identification tasks. The company is also working on a next-gen Universal model, trained on over 10M hours of voice data using Google's TPU chips. AssemblyAI serves over 25 million inference calls and processes over 10TB of voice data daily for clients like Fireflies.ai, Veed, TypeForm, Close, Loop Media, and CallRail. The new funding will support research plans, model development, training compute, market expansion, and team building efforts.

The increasing demand for video editing platforms has led content creators, including digital marketers and YouTubers, to utilize Speech AI models. These models enhance the efficiency of tasks like adding subtitles, identifying key topics, detecting profanity, moderating content, creating highlights, generating transcripts, offering insights, and enabling faster distribution of content on social channels. By leveraging these features, creators can automate tedious tasks while focusing more on creative aspects of video production, leading to improved engagement with their audience.

This week's update includes an improved hold music detection feature, a tutorial for LeMUR applications that utilize LLMs with voice data, blog posts on how to extract insights from customer calls using LLMs and Python, integrating spoken audio into LangChain.js, and analyzing conversations with AI. Additionally, learn about running LLMs locally, converting Hindi speech to text with Python, and accessing the free Playground for conversation analysis.