July 2024 Summaries

This week's update introduces PII Redaction support for an additional 47 languages, allowing users to identify and remove personal data from transcripts or "beep out" sensitive information in audio files. The Entity Detection model has also been upgraded with the addition of 16 new entity types, bringing the total to 44 types with a 99% accuracy rate in major languages. Additionally, users can learn how to use Claude 3.5 Sonnet with audio data, understand Microsoft's Florence-2 foundational image model, and create real-time language translation services using AssemblyAI and DeepL in JavaScript through the latest blog posts.

Anthropic's Claude 3.5 Sonnet sets new industry benchmarks for many LLM tasks, excelling in complex coding and nuanced literary analysis. This tutorial demonstrates how to use Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Haiku with audio or video files in Python using the LeMUR framework from AssemblyAI. The pipeline allows users to transcribe audio data, apply language models to speech data, and generate summaries, action items, or answers to questions about the audio content. Users can access all Claude 3 models through the AssemblyAI platform at no additional cost.

AssemblyAI has announced updates to its PII Text Redaction and Entity Detection features, expanding language support for the former to 47 languages and adding 16 new entity types to the latter for a total of 44 available. These enhancements aim to provide users with more power and control in protecting sensitive information while ensuring privacy and security. The PII Text Redaction feature now supports 13 EU-based languages, meeting regional data residency requirements. Entity Detection allows users to automatically identify and categorize key information in their transcripts, providing detailed entity lists and timestamps.

When choosing a Speech-to-Text provider, it is crucial to consider the security of your data. Confidentiality, integrity, and availability are fundamental aspects of data security, known as the "security triad." Developers should ask their API providers about defense in depth, industry standard frameworks, code-level controls, technical controls supporting data security, and training for developers. Key technical controls include encryption (end-to-end, AES, TLS), malware prevention, role-based access control, handling of sensitive data, transparent data handling policies, and regular security audits and updates. Strong security practices help build trust with customers and maintain the viability of a business.

Microsoft's new large vision model (LVM), Florence-2, is a significant step towards the goal of a unified vision model. It demonstrates impressive results with a compact, parameter-efficient model and can perform a wide variety of image-language tasks such as captioning, optical character recognition, object detection, region detection, region segmentation, vocabulary segmentation, and more. Florence-2 follows the "playbook" of large language models (LLMs) research by building on top of other recent vision research to learn general representations that are useful for many tasks. It is designed in a simple way - to take in textual prompts (in addition to the image being processed), and generate textual results. The architecture unifies the way diverse types of information, such as masked contours, locations, etc., are input to the model, permitting a unified training procedure and easy extension to other tasks without the need for architectural modifications.

Anthropic has introduced Claude 3 model family into LeMUR framework, offering users advanced AI capabilities for audio data analysis. Four new models are now available: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku. Each model offers unique features such as complex analysis, multi-step tasks, enterprise workloads, and near-instant responses. Users can refer to the prompting guide and LeMUR quick start guide for implementation and best practices.

This tutorial demonstrates how to build a web application in Node.js that uses AssemblyAI's Streaming Speech-to-Text API for real-time transcription and DeepL's API for translation into different languages. The app consists of two parts: backend and frontend. The backend is responsible for securely handling the AssemblyAI and DeepL API keys, while the frontend captures audio from the microphone and sends it to the backend for processing. The final transcription is then translated using DeepL's API and displayed on the screen. This application can be useful for non-English speakers visiting English-speaking countries or for any other real-time translation needs.

AssemblyAI's LeMUR framework leverages Large Language Models (LLMs) to understand recognized speech and generate content, summaries, and answers from audio data. With the addition of the Claude 3 model family, users can access state-of-the-art LLMs for various tasks. LeMUR now supports four new Claude 3 models: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku, each offering unique trade-offs to fine-tune Speech AI capabilities for optimal performance, speed, and cost-efficiency. Users can input their own prompts into LeMUR to control the insights and outputs they get from their audio data. AssemblyAI's pricing matches Anthropic's Claude 3 pricing, giving users access to the newest and most powerful LLMs alongside industry-leading speech AI capabilities at no additional cost.

In this tutorial, you'll learn how to build a video playback application that transcribes the audio of an uploaded video file using the AssemblyAI API and then translates the subtitles into different languages using DeepL. Here are the steps involved in building this application: 1. Set up your development environment by installing Go, Git, and Docker. 2. Create a new directory for your project and initialize it with `go mod init`. 3. Add the required dependencies to your go.mod file. 4. Create a new file called main.go and import the necessary packages. 5. Set up environment variables using the .env file. 6. Define the Job struct, which represents an AssemblyAI job. 7. Implement the /jobs route, which uploads the video file to the server and creates a transcription job on AssemblyAI. 8. Add the necessary routes for polling the status of the transcription job and downloading the subtitles when the transcription is complete. 9. Create a new view called job.html that displays the video with captions. 10. Implement the /translate route, which translates the subtitles using DeepL's API. 11. Modify the job.html file to include a language selector and event listeners for handling translation requests. By following these steps, you can create a simple video playback application that transcribes audio and generates subtitles in different languages on demand. This application can be further extended by adding features like speaker diarization, profanity filtering, custom spelling, and custom vocabulary using AssemblyAI's API.

This weekly update provides information on new product features, tutorials, and community updates from the company. It introduces guides on using Large Language Models (LLMs) with LeMUR for asking questions, summarizing audio data, extracting and generating content from voice data. The update also highlights AssemblyAI's integrations with LangChain, LlamaIndex, and Twilio to enhance call transcription and build searchable audio archives. Additionally, it features blog posts on creating multi-lingual subtitles using AssemblyAI and DeepL, building an AI-powered video conferencing app with Next.js and Stream, and detecting hotwords with streaming speech-to-text and Go. Lastly, the update includes trending YouTube tutorials for speaker-based subtitles, voice translation, and creating an AI chatbot in Java.

In this tutorial, you will learn how to build a Next.js video conferencing app that supports video calls with live transcriptions and an LLM-powered meeting assistant. The app uses the Stream Video SDK for video call functionality, AssemblyAI for real-time transcription and LLM functionality, and Node.js 18.17 or later to build with Next.js. The final app allows users to start a video call, transcribe conversations in real-time, and ask questions during the call that are answered by an AI assistant.