August 2024 Summaries

This weekly update highlights the latest product features, tutorials, and community updates from AssemblyAI. The company invites founders, product, and engineering leaders to share their AI insights for a chance to earn a $20 gift card. Additionally, they have introduced major updates to their Automatic Language Detection (ALD) model, offering industry-leading accuracy in 17 languages including Chinese, Finnish, and Hindi. The ALD now supports customizable confidence thresholds, allowing users to set minimum confidence levels for language detection and manage low-confidence cases.

Speech-to-text technology is a sophisticated system that converts spoken words into written text. It relies on linguistics, computer science, and artificial intelligence to function. This technology has found its way into several applications across various industries and personal use cases. Some popular speech-to-text tools include AssemblyAI, Google Cloud Speech-to-Text, and Amazon Transcribe. The future of this technology holds improvements in accuracy, new AI-powered features like emotion detection and intent recognition, and emerging applications across different sectors.

This tutorial guides you through enhancing virtual meetings with AI by extracting audio from a Zoom call, transcribing it using AssemblyAI, and analyzing the transcription with AssemblyAI's LeMUR API and audio intelligence models. The process involves setting up a development environment, installing required dependencies, retrieving an AssemblyAI API key, streaming audio to a custom media server, parsing the audio from the RTMP stream, transcribing it using AssemblyAI, and analyzing the transcription with various tasks such as summarization, sentiment analysis, and PII redaction. The code used in this tutorial can be found on GitHub.

AssemblyAI has updated its Automatic Language Detection (ALD) model, increasing accuracy and expanding language support from seven to seventeen languages in the Best Tier. The improved ALD model is benchmarked against four leading market providers, demonstrating industry-leading accuracy in detecting spoken languages. This update allows users to build multilingual applications with higher accuracy and more control, providing better insights and experiences for customers. AssemblyAI's API supports 17 languages in the Best Tier and 99 in Nano, simplifying multilingual application development and reducing maintenance time. The ALD model also introduces customizable confidence thresholds, allowing users to set minimum confidence levels for language detection and ensuring only high-certainty transcriptions are processed.

This weekly update highlights new product features, tutorials, and community updates from AssemblyAI. The feature spotlight is on Streaming Speech-to-Text (STT), which allows for live audio stream transcription with high accuracy and low latency. Applications built using this technology include real-time medical transcription analysis, voice bot integration with Meta's Llama 3, and AI-powered voice assistants for call centers, customer support, and virtual receptionist applications. Additionally, the blog offers tutorials on hotword detection with Streaming STT in Go, scam call detection using LeMUR and Twilio, and building an AI-powered video conferencing app with Next.js and Stream. The YouTube tutorials cover real-time medical transcription analysis using Python, summarizing YouTube reviews with LLMs, and building a chatbot with Claude 3.5 Sonnet and audio data in Python.

Many consumers find it frustrating to deal with multiple customer service representatives, as they often have to repeat their problems or questions. This can lead to a negative perception of the company and may cause customers to switch brands. To improve customer satisfaction, companies are turning to conversation intelligence platforms that use Voice of Customer data and AI analysis. These platforms help businesses understand their customers' needs and experiences better, leading to more personalized interactions and improved engagement in the long term. Speech AI models can capture conversations, transcribe them into text, and perform sophisticated analyses on conversational data, providing valuable insights for companies. Conversation intelligence tools can improve customer service, optimize sales, enhance training and development, gather product feedback, refine marketing strategies, boost customer retention, ensure quality assurance, inform competitive analysis, provide real-time insights, and create personalized customer experiences. By combining AI with a human touch, companies can better understand their customers' needs and challenges, leading to more satisfied customers in the long run.

The text discusses various decoding strategies used in Language Models (LLMs) to generate coherent and contextually appropriate text. It highlights the distinction between next-word predictors and text generators, emphasizing that LLMs don't always output the most probable next word iteratively but employ different decoding strategies for text generation. The article delves into deterministic methods like Greedy Search and Beam Search, stochastic methods such as Top-k, Top-p (Nucleus Sampling), and Temperature Sampling, and novel methods based on information theory like Typical Sampling. It also discusses Speculative Sampling, a technique to enhance LLM inference speed by generating multiple tokens per model pass without changes to the final output. The text concludes with an outlook for future research in this area.

In June 2024, AssemblyAI updated its Speaker Diarization model to be 13% more accurate and added support for five additional languages. This improvement helps users accurately identify who is speaking in audio recordings, making it easier to analyze conversations in more languages. The Speaker Diarization feature can be applied to distinguish between speakers in audio projects and can also infer speaker names using LeMUR. It enhances audio analysis by accurately identifying and differentiating speakers, improving transcripts, enabling searchable audio, and providing better training for language-based AI tools. Additionally, AssemblyAI offers tutorials on generating subtitles with Zapier, detecting scam calls using Go with LeMUR and Twilio, content moderation on audio files with Python, building a web app to summarize YouTube reviews with LLMs, real-time speech-to-text in Java, and live speech-to-text transcription in Google Docs using Python.

AssemblyAI has released a Ruby SDK, making it easier to use their Speech AI models with Ruby programming language. The SDK allows users to transcribe audio files, analyze audio using intelligence models, and apply LLMs (Large Language Models) to audio data using LeMUR. Three examples are provided: transcribing an audio file from a URL or local file, using LeMUR for LLM applications on voice data, and utilizing audio intelligence models for sentiment analysis. The Ruby SDK can be found in the GitHub repository with installation instructions and documentation available.

Customer success (CS) and support teams are crucial for companies, especially those dealing with technical products like AI. These teams not only solve problems but also build relationships, foster loyalty, and drive continued business growth. The quality of customer support can be a deciding factor when customers choose a service. CS and support teams need to be proactive in understanding and anticipating customer needs, questions, or pain points to help them achieve successful AI implementation, launch, and continued growth. Great customer success teams consistently go above and beyond what customers expect, understand the context behind customer questions, and focus on enabling and unblocking customers. Companies that invest in understanding their customers' needs and proactively assisting them will always stand out.

The accuracy of Speech-to-Text (STT) systems is highly dependent on the quality of audio input. Selecting the appropriate audio file format is crucial, as it directly impacts how accurately the system can interpret and transcribe spoken words. Key considerations for choosing an audio format include sound quality, file size, compatibility with STT software, sample rate, bit depth, and compression. The most commonly used audio formats for Speech-to-Text are WAV, FLAC, MP3, AAC, and M4A. While post-processing can sometimes improve transcription accuracy, it is essential to focus on capturing high-quality recordings from the start and apply minimal, targeted enhancements. For video files, choosing the right format is equally important, as video containers like MP4, MOV, AVI, and MKV impact both audio quality and file size. Ultimately, the right format for your Speech-to-Text project will depend on the specific requirements of your application, the quality of the original audio recording, and the capabilities of the STT system you're using.

LeMUR has introduced two new endpoints, Question & Answer and Summary, for Claude 3 models including Opus, 3.5 Sonnet, 3 Sonnet, and Haiku. These additional features complement the existing Task endpoint. AssemblyAI's Zapier integration version 2.0 allows users to create more complex, multi-step Zaps with platforms like Notion, Google Docs, Twilio, and YouTube. New events have been introduced for transcript retrieval, subtitle generation, segmenting transcripts into paragraphs or sentences, and accessing PII-redacted audio files.

The accuracy of Speech-to-Text (STT) systems is highly dependent on the quality of audio input. Selecting the appropriate audio file format is crucial, as it directly impacts how accurately the system can interpret and transcribe spoken words. Key considerations for choosing an audio format include sound quality, file size, compatibility with STT software, sample rate, bit depth, and compression. The most commonly used audio formats for Speech-to-Text are WAV, FLAC, MP3, AAC, and M4A. While post-processing can sometimes improve transcription accuracy, it is essential to focus on capturing high-quality recordings from the start and apply minimal, targeted enhancements. For video files, choosing the right format is equally important, as video containers like MP4, MOV, AVI, and MKV impact both audio quality and file size. Ultimately, the right format for your Speech-to-Text project will depend on the specific requirements of your application, the quality of the original audio recording, and the capabilities of the STT system you're using.

The new version of the AssemblyAI app for Zapier now offers more options and features, including an updated Transcribe event with additional settings such as language code and punctuation casing. Additionally, five new events have been added to the app. However, LeMUR support is not yet available on the Zapier platform due to its lack of necessary features. Users can try out the new AssemblyAI app for Zapier by following a guide or tutorial provided by the company.

In this tutorial, learn how to transcribe a video using the AssemblyAI app for Zapier and generate SRT subtitles for it. The process involves creating a new Zap, configuring triggers, transcribing audio, generating subtitles, and uploading them to Google Drive. To follow along, you need basic experience with creating Zaps on Zapier, an upgraded Zapier plan, a Chromium-based browser, and the Zapier Chrome extension. The tutorial also provides sample data for testing purposes.

The artificial intelligence industry is predicted to become a trillion-dollar market in less than a decade, transforming the way we learn, work, and interact with technology and people daily. However, there's little guidance available on how to evaluate AI systems to choose the best option for specific needs. AssemblyAI believes that AI systems are only as good as the benchmarks and evaluations they are measured against. Consistent benchmarks ensure that AI models perform at human-level standards, providing a clear, unbiased yardstick for comparing different AI solutions. Independent third-party organizations are necessary to conduct evaluations and benchmarks of AI systems, ensuring impartiality and scientific integrity. While there are only a few objective third-party organizations evaluating AI systems right now, more will emerge soon.

AssemblyAI's Python SDK has reached over 100,000 monthly downloads, demonstrating the growing interest in its features such as speaker diarization, entity detection, and LLMs for audio files. The company also offers SDKs for JavaScript/TypeScript, Go, Java, and Ruby, with more languages on the horizon. Additionally, AssemblyAI's blog provides tutorials on using their services with various applications like video conferencing and real-time language translation.