October 2025 Summaries

Since the release of DeepSeek-R1 in January 2025, there has been a notable rise in the use of reinforcement learning based on verifiable rewards (RLVR), sparking interest in its application for enhancing products and academic research, especially in multimodal contexts. RLVR stands out because it allows models to explore and exploit actions for rewards, which is particularly beneficial when traditional differentiable loss functions are hard to define, such as in tasks requiring reasoning or dynamic decision-making. However, challenges like entropy collapse, where models become overly deterministic, highlight the need for a balanced exploration-exploitation strategy. In multimodal settings, successful RL applications often require a combination of offline RL training data, cold-start data from supervised fine-tuning, and the careful scaffolding of training data. This scaffolding ensures stable training and prevents common pitfalls like reward hacking and training instability. The text discusses various approaches to generating data for multimodal RL, including leveraging human perception with automated verification, model-generated descriptions for video prediction, and chaining specialized models for geometric tasks, while emphasizing the importance of curriculum learning to progressively increase task complexity. The iterative process of refining data acquisition and integrating human feedback is crucial, not only for starting the RL process but also for improving and maintaining the system’s performance over time.

Segment Anything Model 3 (SAM 3), a paper under review for ICLR 2026, represents a significant advancement in computer vision by expanding the capabilities of its predecessors through Promptable Concept Segmentation (PCS). Unlike SAM 1 and 2, which relied on geometric prompts, SAM 3 can segment all instances of a user-defined concept across images and video sequences, integrating image, video, and text into a unified architecture. This open-vocabulary model can identify concepts from noun phrases or visual exemplars, making it invaluable for applications in robotics, scientific imaging, and AI data pipelines. Its architecture combines a DETR-based object detector and a memory-based tracker to enhance detection and tracking capabilities. SAM 3 also introduces a new data engine combining human expertise with AI to improve annotation scalability. It outperforms previous benchmarks in open-vocabulary segmentation and tracking, though it faces challenges in fine-grained categories and long expressions. If released as open-source, SAM 3 could revolutionize vision-language segmentation, concept-level search, and interactive annotation, paving the way for more advanced AI infrastructure solutions.

Autonomous vehicles rely on accurately labeled multimodal data, such as video, LiDAR, radar, and sensor inputs, to safely perceive their surroundings, making data labeling platforms crucial in their development. Encord is highlighted as a leading platform in 2025, offering comprehensive multimodal support, AI-assisted automation, and robust quality assurance, which is essential for precise annotations across complex datasets. The platform's capabilities include 2D, 3D, LiDAR, radar, and video support, along with SAM2 integration for automation and enterprise-grade compliance with SOC 2 and GDPR standards. Other notable platforms like Scale AI and Segments.ai also provide high-throughput labeling and efficient annotation tools, essential for scalable data handling and real-time decision-making in autonomous systems. The right choice of platform ensures accelerated model training cycles without sacrificing precision, with Encord being particularly distinguished for its end-to-end solutions in the autonomous vehicle sector.

Data labeling platforms are crucial in the development of AI models across various applications, such as smart cities, logistics, sports, robotics, and generative AI, as they transform unstructured data into formats suitable for training. The precision of data labeling significantly impacts model accuracy, with high-quality annotations leading to more reliable AI outcomes. The article compares top AI data annotation platforms like Encord, Scale AI, and others, highlighting their ability to handle diverse data types and incorporate quality assurance and AI-assisted labeling techniques. Choosing the right platform is essential to ensure that AI projects are built on precise, scalable, and high-quality datasets, which can make the difference between underperforming models and those that deliver accurate results.

Henry, a Graduate Associate Software Engineer at Encord, shares his journey from studying Chemical Engineering to joining the fast-paced startup world to build AI tools. After an initial plan to do a ski season fell through, he attended a unique coding bootcamp that paid participants to learn, which led to starting his own company before realizing he preferred a more structured work-life balance. Encord's interview process stood out for its collaborative nature, and since joining, Henry has enjoyed the supportive and socially vibrant environment that allows for autonomy and rapid learning. He works on projects related to enterprise user management, product engineering, and AI integrations, emphasizing the importance of building and shipping real-world solutions. Henry advises new graduates to demonstrate initiative and problem-solving skills by creating projects themselves, as startups value curiosity and creativity. Encord supports early-career engineers with opportunities for significant impact and recognition, encouraging ownership and visibility of work without micromanagement.

Over the past few months, the Encord machine learning team has developed what they claim to be the world's largest open-source multimodal dataset, designed to support the development of models that integrate text, images, video, audio, and 3D point clouds. This dataset aims to facilitate advancements in multimodal AI by providing a clean and extensive resource for open-source development. The process involved sourcing data from multiple modalities, using retrieval models to align the data, and enhancing data quality through human annotation. They also created a retrieval model capable of embedding all modalities into a common space, evaluated through public benchmarks and a newly built dataset for audio-point cloud embeddings. A baseline retrieval model was trained, demonstrating that high-quality data can outperform larger parameter models in cross-modal retrieval tasks. The Encord team hopes that sharing their methodology will aid others in constructing similar datasets and furthering multimodal AI innovation.

Vialytics has transformed inconsistent road imagery into AI-powered solutions for smart cities by developing a scalable, unified data platform to handle the variability and challenges of real-world data. Initially constrained by limited resources, vialytics relied on self-hosted annotation tools and local storage, which quickly became unsustainable as their dataset grew. By adopting Encord, they centralized their data management and annotation workflow, improving data accessibility, visibility, and quality control through automated review processes and flexible annotation workflows. This enabled vialytics to manage diverse and unpredictable data, ensuring consistent model performance across different geographies and allowing for faster iteration cycles. The unified AI data stack allowed vialytics to reduce annotation overhead, focus on edge cases, and deliver reliable outputs, ultimately supporting the development of robust computer vision models essential for smart city infrastructure. Looking forward, vialytics plans to integrate more diverse data types, including sensor data, to enhance their ability to interpret infrastructure conditions, emphasizing the importance of a scalable AI data stack for effective urban infrastructure management.

In a recent masterclass recap by Encord, the challenges of deploying computer vision models in production were explored, focusing on why these models often fail despite accurate training. The session highlighted that failures frequently arise from edge cases—instances in datasets that deviate significantly from typical examples, such as low-light images or unusual object orientations. These edge cases can lead to performance drops, which are exacerbated by issues like labeling errors, poor data quality, data drift, and static models. Strategies for identifying and addressing these problems include segmenting datasets by metadata to track performance dips, visualizing patterns of failure through embedding plots, and leveraging metric correlations to pinpoint problematic data features. The case study of SwingVision, an AI platform analyzing tennis matches, demonstrated how detecting and curating diverse training samples helped improve model robustness. The session emphasized that labeling every piece of data is inefficient; instead, focusing on high-value samples can lead to better resource allocation. Encord's platform facilitates the active learning cycle, enabling continuous improvement through strategic data curation and evaluation, which ultimately enhances model reliability in real-world conditions by adopting smarter data practices.