June 2024 Summaries

The Open AI Spring Update Event showcased significant advancements in Large Language Models (LLMs), particularly GPT-4, which can understand emotion and be interrupted. However, to truly be valuable, these models need to work reliably over challenging real-world internet conditions across various mobile devices and networks. Agora has been focused on building infrastructure and end-to-end technologies to deliver the real-time internet, optimizing performance for natural conversations despite wireless last mile challenges. Agora's infrastructure is used by thousands of app developers globally, powering reliable real-time communications on a large scale.

This guide outlines how to combine AI with Agora to build a video call app that uses AI to summarize calls. The app starts by creating a simple video call using Agora, and then implements real-time transcription of the call using Agora's Real-Time Transcription service. The transcription is stored on the client side until it can be used to prompt an AI model to generate a summary. The guide covers how to set up the Real-Time Transcription service, decode the serialized data, store the transcript, and use it to prompt the AI model for a summary.

The Agora Web SDK provides a powerful set of tools for real-time engagement, allowing developers to easily embed live voice and video functionalities into web applications. The SDK offers convenient APIs that can automatically embed a video element within a specified `<div/>`, but also allows for complete customization of video elements using JavaScript/TypeScript. To integrate the Agora Web SDK, developers need to add it to their project by running the command `npm install agora-rtc-sdk-ng`. Once installed, they can initialize the camera and mic using the `.createMicrophoneAndCameraTracks()` function and set up a custom video element using the `setupVideoElement()` function. The video element is created using a MediaStream object, which acts as a wrapper for the camera track, allowing it to be passed off to the `<video/>` element. By following this guide, developers can manipulate audio and video tracks, customize video player controls, and integrate interactive elements to enhance user engagement.

In today's digital-first business landscape, real-time communication via voice and video is crucial for immediate and impactful user engagement across industries. WebRTC is an open-source protocol that enables developers to build robust voice, video, and data-sharing solutions, but setting up a DIY WebRTC backend solution can be complex due to technical hurdles like buffering, latency, and optimizing video quality. Businesses face challenges in maintaining stable and high-bandwidth connections, as internet routing is difficult to control, and ISPs may implement traffic shaping or throttling. Agora tackles this challenge by deploying Selective Forwarding Units worldwide and using machine learning to optimize routing. The selection of a video codec requires balancing factors like video quality, compression efficiency, and compatibility, with Agora's dynamic adjustment of codecs based on CPU load and network conditions ensuring high-quality video streaming. Real-Time Communication applications rely on complex infrastructure of servers, including application servers, signaling servers, NAT traversal servers, and media servers, posing significant cost and resource challenges. Agora's managed network of servers strategically deployed worldwide minimizes distance and latency, delivering superior quality experiences.

As younger audiences spend more time viewing content on mobile apps rather than traditional TV and streaming, media and entertainment brands need to enhance their streaming experience by embracing interactivity and mobile viewing. To achieve this, they must add features such as multi-camera live streams, user-generated content, personalized avatar overlays, and real-time functionality, which require ultra-low latency delivery and synchronization of data in real time. These interactive experiences not only keep viewers engaged for longer periods but also provide valuable first-party data from social interactions that can be analyzed by generative AI to uncover new trends and give companies a competitive edge. To enable these features, media brands need to move beyond traditional content delivery methods and adopt Agora's Software-Defined Real-Time Network (SD-RTN) technology, which ensures ultra-low latency and real-time synchronization of data for seamless interactive user experiences.

Today, content creators and their audiences are looking for more than standard live streaming functionality, as enhanced interactive features and multi-dimensional communication options within the live stream have become critical to driving engaging experiences. CelebPrime, a flagship product of Fanory, is innovating on the standard model with advanced functionality, including real-time AR effects that add Augmented Reality (AR) to create more interactive and engaging experiences. Creators can express themselves uniquely using AR effects for enhanced storytelling or humor, while face detection ensures adherence to community guidelines through real-time content monitoring. The platform also offers multi-call functionality powered by Agora's robust APIs, enabling fans to engage in live discussions and fostering a dynamic environment. By strategically implementing advanced features and prioritizing both the user and creator experience, CelebPrime has solidified its position as a leader in the live streaming industry.