September 2023 Summaries

The Daily developer platform is powering audio and video experiences for millions of people worldwide. The company has introduced new toolkits, components of its global infrastructure, and AI-focused partnerships as part of its AI Week series. One focus area is telehealth, where AI can help expand access to care and improve patient outcomes by automating clinical notes generation. This process involves capturing audio, transcribing it, processing and transforming the transcription, validating and storing the output. The pipeline requires data privacy, accuracy, reliability, flexibility, and has been extended with HIPAA-compliant infrastructure and partnerships. Daily has also partnered with Deepgram and ScienceIO to deliver high-quality clinical notes generation using GPT-4 and Llama-2 models. The company plans to continue releasing new APIs and features for video analytics, composition, and editing.

Daily's developer platform powers audio and video experiences for millions of people worldwide, using APIs and client SDKs to build features into applications and websites. The company's AI Week series explores how developers can combine real-time video with AI, focusing on content moderation as a key use case. By combining Daily's daily-python SDK with ActiveFence's robust content moderation APIs, developers can create flexible and powerful solutions for identifying unwanted behavior, safeguarding users, and meeting industry standards. This approach leverages AI-powered detection to analyze text, images, audio, and video in real-time, providing risk scores and indicating unwanted activity. Daily aims to provide developers with ergonomic APIs that allow for maximum customizability, while supporting content moderation for every platform with great resources and competitive pricing.

Daily has introduced a Python SDK, named daily-python, designed to integrate AI into real-time WebRTC video and audio sessions, as part of their AI Week initiatives. The SDK allows developers to build applications that leverage AI models and services, enabling features such as speech-to-speech translation, creation of virtual characters, content moderation, and interactive gaming within video calls. Built on a shared Rust and C++ codebase, daily-python provides high-performance media processing and runs efficiently in both server-side and interactive environments, supporting platforms like Microsoft Azure, AWS Lambda, and desktop machines. The SDK is intended to work seamlessly with Daily's existing APIs and is designed to operate as a "headless" bot within a Daily room, capable of processing video, audio, and events streams. As AI technology rapidly evolves, Daily anticipates innovative applications emerging from developers using daily-python and encourages engagement through their blog, Discord, and community events.

Daily, a developer platform for audio and video experiences, is expanding its offerings by integrating AI into its services. The company recently launched AI Week, featuring new APIs, toolkits, and infrastructure components designed to enhance video and audio applications with AI capabilities. These developments include the AI-powered Clinical Notes API for telehealth, a Python SDK for AI-enabled WebRTC use cases, and an Automatic Video Highlights toolkit. Daily's focus is on enabling developers to leverage AI for real-time video and audio features, such as content moderation, real-time translation, and automated video editing. The platform aims to help developers utilize AI to extract more value and insights from existing data, building on its established real-time video capabilities. Daily's efforts reflect its commitment to supporting innovative use cases and expanding the possibilities of combining AI with video and audio technologies.

Daily provides global infrastructure and developer SDKs for high-quality video and audio communications, serving telehealth customers since its early days and offering several industry-first features such as HIPAA-compliant APIs, one-click video calls, and an integrated customer support dashboard. The company is now announcing a new feature: clinical notes generation seamlessly integrated into telehealth developer APIs, powered by advanced AI tools that can help healthcare providers save time and reduce clinician burnout. This feature uses large language models to generate transcripts from recordings of any telehealth session, generate AI-powered SOAP notes with a single API call, deploy a polished UX for care providers, and allow developers to rapidly explore and ship custom LLM capabilities and structured medical data workflows on HIPAA-compliant infrastructure. The new clinical notes APIs are part of Daily's existing HIPAA offering and support several ways to build telehealth solutions, including using the company's HIPAA-compliant Video SDKs, adding secure one-click video to web applications, and deploying AI-generated clinical notes in a variety of ways.

This is an interesting project by the front-end engineers at Daily, where they integrated a GPT-4-powered meeting assistant into their video app using React and JavaScript. The team created a chatbot that can transcribe conversations in real-time, send prompts to OpenAI, receive responses, and display them in a private chat message. They also experimented with storing transcripts in the app's state for synchronization purposes. However, they acknowledge limitations such as the 32k token limit of GPT-4 and plan to address these issues by providing summaries or storing transcripts as vector embeddings. The project showcases the potential of AI-powered meeting assistants and highlights the Daily team's ability to quickly develop and integrate new features into their platform.

Liza Shulyayeva discusses the implementation of a new feature in Daily's JavaScript client SDK, which allows monitoring of audio levels for both local and remote participants in video calls. The feature is demonstrated in "Code of Daily: Modern Wordfare," a social game similar to Codewords. The audio level monitoring involves methods like `startLocalAudioLevelObserver` and `startRemoteParticipantsAudioLevelObserver`, which track audio levels at specified intervals and emit events with audio data. The implementation involves updating the visual representation of participants' audio levels by adjusting the opacity of a border around each participant's tile, creating a smooth transition effect as players speak. This capability opens up possibilities for developers to create engaging features using real-time audio data.

This latest update from Daily's HTTP Live Streaming capabilities enhances developer flexibility in creating high-quality live streaming experiences with customizable multi-layer HLS output at 1080p, adjustable bitrate options, and the ability to stream directly to Amazon S3 or distribute with a CDN of choice. The new features also include previewing 'trick play' streams, synchronizing UI events with content playback using time-code metadata, and targeting multiple destinations with just one line of code. These enhancements aim to provide developers with unparalleled flexibility in broadcasting real-time WebRTC video content across multiple platforms, while ensuring they retain control over designing experiences and distributing content.

The author of this text is testing external API calls in Go using the httptest package from Go's standard library. The test uses a table-driven approach with multiple test cases to verify that the code handles data returned by Daily's REST API correctly. The test server is spun up and destroyed for each test case, simulating the call to Daily's API. The author checks if the error returned matches what they expect, then compares the actual values of the participants returned with the expected values. The test also covers failure responses from Daily's API, such as internal server errors.

The implementation focuses on creating a video call room and retrieving the list of attendants before the participant joins, allowing for a more organized and comfortable experience. The app uses Daily's Client SDK for JavaScript to create a pre-join lobby and display the number of participants already in the call. When a participant joins, they can see who else is in the call through Prebuilt's own UI. The server-side implementation involves making an HTTP request to Daily's REST API to retrieve the presence data, which is then unmarshalled into a JSON format and displayed on the client-side. The code showcases how to implement this feature using JavaScript and Go, with the relevant Daily REST API endpoints remaining the same regardless of language.