March 2024 Summaries

Multimodal AI combines data types such as text, images, audio, video, and sensor data to create systems that interpret information more comprehensively than single-modality models, with potential uses in healthcare, accessibility, content creation, augmented reality, and conversational systems. Its foundations include converting each modality into useful representations and integrating them through early, intermediate, or late fusion, while major architectures include transformer-based models, generative models such as VAEs and GANs, and multimodal language models capable of tasks like image captioning, visual question answering, and text-to-image generation. A case study describes fine-tuning the IDEFICS 9B model on paired Pokémon Go screenshots and descriptions to generate narratives about augmented-reality gameplay, illustrating how visual and textual inputs can support more interactive experiences. Key technical challenges remain, including aligning heterogeneous data, accommodating missing modalities, and managing the greater computational complexity of multimodal training and deployment.

Web developers can improve cross-browser and cross-device audio playback by favoring the HTML audio element over the Web Audio API, particularly because autoplay restrictions are stringent on platforms such as iOS. The discussion recommends initiating playback through user actions to comply with browser policies, using audio sprites and compressed formats to reduce requests and loading time, lazy-loading files, providing accessible volume controls, remembering preferences where appropriate, and regularly testing across browsers, devices, and screen sizes with feature detection and responsive design. It also presents react-nowplaying, a React package intended to simplify these concerns by providing a context-based audio player that can play audio from URLs or Blob data and exposes native playback controls and events.

Deepgram API integrations can encounter HTTP 429 errors when applications exceed request-rate limits, signaling that requests should be slowed to protect shared service capacity and avoid disruptions or temporary API-key blocks. The guidance explains how Node.js and Python applications can identify 429 responses, retry requests with exponential delays that increase after each failure, and set maximum retry limits while handling other HTTP errors separately. It also recommends adding randomized jitter to retry delays to prevent many application instances from retrying simultaneously, dynamically optimizing request rates when rate-limit information is available, and logging API usage and 429 incidents to improve application behavior over time.

AI tools can help startups offset limited staffing and funding by automating routine work, improving decision-making, and supporting marketing, customer engagement, and operations. The overview highlights Jasper for brand-consistent marketing content and campaign planning, Notion AI and Mem for organizing knowledge and answering questions from internal information, and Socialbee for creating, scheduling, and managing social media content. It also presents Aible for data analysis and custom generative AI applications, Lavender for improving email effectiveness, and HubSpot’s ChatSpot for CRM-based content creation, customer engagement, and task management. Claude is described as a conversational AI capable of handling complex tasks such as document summarization, image analysis, and translation, while Deepgram provides speech-to-text and text-to-speech APIs for voice-enabled applications. Together, these tools are positioned as ways for small businesses to increase productivity and compete more effectively with larger organizations, while still requiring human judgment.

Deepgram announced its sponsorship and booth presence at NVIDIA GTC 2024, where it plans to demonstrate its voice AI platform, including the Nova-2 speech-to-text model and Aura text-to-speech model for real-time conversational applications. The company highlights low-latency transcription and speech generation, developer use cases such as virtual agents, patient intake, customer support, and voice ordering, and an open-source demo with free introductory credits for Aura. Visitors to booth #1719 can view technology demonstrations, try the voice-controlled Yappybird game for a chance to win a Rabbit R1 device, and meet the RoboDoge robot mascot. Deepgram is also co-hosting an AI Mixer with Clarifai on March 19 to discuss their partnership, which combines Deepgram’s speech models with Clarifai’s platform for building and deploying AI applications involving large language models and unstructured data.

AI tools have become widely used for research, content creation, creative work, and business automation, though the large number of available products can make selecting the right one difficult. The article highlights three specialized platforms that use AI to improve customer-facing operations: UpdateAI, which analyzes online meetings to produce summaries, action items, speaker talk ratios, transcripts, and meeting preparation materials; Elerian AI, which enables companies to build low-code, human-like voice bots that support varied accents and dialects, customer authentication, profile access, scheduling, and sentiment analysis; and Nytro.ai, which evaluates sales representatives’ pitch recordings to support onboarding, training, and ongoing performance development. Each platform illustrates how AI can reduce administrative work, improve customer interactions, and provide targeted coaching or operational insights for business teams.

Deepgram has launched Aura, a generally available text-to-speech API designed for low-latency, high-throughput conversational AI agents and applications. Aura provides 12 English voices, natural conversational features such as pauses, breaths, hesitation sounds, and contextual tone changes, and average response times below 250 milliseconds, with pricing beginning at $0.015 per 1,000 characters. Combined with Deepgram’s Nova-2 speech-to-text service and external language models, Aura completes the company’s Voice AI platform across listening, reasoning, and speaking functions. Deepgram positions Aura for real-time interactions such as customer support, appointments, ordering, and voicebots, distinguishing these workloads from quality-focused content production such as audiobooks and advertising. Early users including Humach, Vapi, and Daily have integrated Aura into production or development platforms, citing its speed, voice quality, and integration with Deepgram’s transcription services. The company also offers an open-source demonstration, API access, $200 in new-user credits, and plans to expand Aura with more lifelike voices, languages, and use cases.

Large language models are first pre-trained on enormous unlabeled text corpora to develop broad language knowledge, then fine-tuned on much smaller task-specific datasets to make them useful for applications such as conversation, question answering, and summarization. Fine-tuning is far less costly than pre-training and commonly begins with supervised fine-tuning, which trains on curated prompt-response pairs using next-token prediction, though results depend heavily on data quality and can risk catastrophic forgetting. Further alignment can use reinforcement learning from human feedback, in which people rank model outputs to train a reward model that guides optimization, or Direct Preference Optimization, which directly learns from preferred and rejected responses without requiring a separate reward model or reinforcement-learning loop. The discussion also highlights parameter-efficient fine-tuning methods such as LoRA, which learns small low-rank weight updates instead of modifying all model parameters, and QLoRA, which adds quantization and memory-management techniques to make large-model fine-tuning feasible on more modest hardware.

Deepgram’s Startup Program, launched in Q2 2023, supports emerging audio AI companies by providing access to its speech technology, credits, technical guidance, and promotional opportunities. The program has grown to include 110 startups across sectors such as healthcare, podcasting, phone assistance, AR/VR, SaaS, education, and media, whose combined usage has reached 10 million minutes and more than $200,000 in credits; 28 startups joined in the first quarter of 2024 alone, using over 2.3 million free minutes. Participants receive early access to releases including the Aura text-to-speech model, direct Slack access to applied engineers, potential joint go-to-market exposure, and educational office hours covering topics such as Rust and AI, real-time streaming, API optimization, VR applications, and text-to-speech.