July 2026 Summaries

Jolia, developed by Raidium, is a 3D CT vision-language foundation model that generates a global [CLS] embedding and 102 named per-organ embeddings from paired chest and abdominal CT scans, without requiring segmentation masks or spatial supervision. FiftyOne, an open-source tool by Voxel51, facilitates the exploration and visualization of these embeddings, allowing for interactive UMAP visualization, per-organ similarity search, and attention heatmap overlays. By using the CT-RATE dataset, the integration of Jolia and FiftyOne demonstrates how per-organ embeddings provide independent views of scans, offering insights into model behavior and interpretability. Jolia's innovative training method, ConQuer, aligns anatomical regions with corresponding report sections, enabling it to achieve state-of-the-art performance in findings classification and report generation. While Jolia and FiftyOne are not intended for clinical diagnosis, they serve as valuable research tools for understanding and improving medical imaging models.

KITScenes Multimodal, developed by the Institute of Measurement and Control Systems at the Karlsruhe Institute of Technology and FZI Research Center for Information Technology, is an extensive European urban driving dataset featuring a dense sensor suite with nine synchronized cameras, seven lidars, and three radars. This dataset aims to enhance the fidelity of autonomous driving datasets by offering high-resolution imagery, detailed lidar point clouds, and comprehensive HD maps annotated in the Lanelet2 format, which provide full topological connectivity. Integrated with the FiftyOne platform, KITScenes Multimodal organizes multiple sensor inputs into single, synchronized samples for more efficient data processing and analysis, making it a valuable tool for developing and testing autonomous driving technologies. The current FiftyOne build serves as a preview with validation scenes from Frankfurt, allowing users to explore the dataset's capabilities and develop data pipelines, though it lacks certain dynamic annotations present in the full corpus.

FiftyOne Enterprise 2.21.0 enhances its data annotation platform with new features like Annotation Workflows, video annotation, and temporal tags to improve team collaboration and streamline the labeling process. The update builds on the open-source FiftyOne 1.18.0 release, adding Enterprise-exclusive features such as multi-stage annotate-and-review pipelines, role-aware task queues, and in-app annotation schemas for consistent labeling. Video annotation is now generally available, allowing users to label object tracks directly in the app with keyframes and interpolation, while temporal tags enable time-based labeling for events. The release also includes quality-of-life improvements like faster app startup, new embeddings, and Transformers 5.x support, making FiftyOne a comprehensive solution for data annotation and management.

The Community Fish Detector is a single-class object detection model designed to identify fish in various aquatic environments, leveraging the comprehensive Community Fish Detection Dataset that contains over 1.9 million images. The dataset, harmonized into a single COCO archive from 17 diverse sources, presents a robust challenge for model evaluation, as it includes environments ranging from murky Danish waters to Australian billabongs. FiftyOne, an open-source tool, enhances this evaluation by providing a visual interface that goes beyond aggregate metrics, allowing users to explore per-source mean Average Precision (mAP) scores and identify domain-specific model weaknesses. It facilitates the detection of annotation errors and unlabeled fish within the dataset, offering a more granular understanding of model performance. This method of evaluation, which involves streaming curated data subsets to avoid downloading the entire dataset, enables users to gain insights into the model’s efficiency across different environments.

CarCrashNet, a collaboration between MIT and the Toyota Research Institute, is an open-source benchmark featuring over 15,000 structural crash simulations, making it the first large-scale dataset of its kind for data-driven crash simulations. This extensive dataset includes simulations for three vehicle models—Dodge Neon, Toyota Yaris, and Chevrolet Silverado—using the open-source solver OpenRadioss and validated against Ansys LS-DYNA and physical crash tests. FiftyOne, an open-source visual AI tool, has been employed to handle the dataset, enabling the exploration and curation of crash videos, static figures, tabular metrics, and learned embeddings. With its ability to create multi-camera group slices and a live benchmark leaderboard, FiftyOne demonstrates its versatility in handling scientific simulation data, even though it was originally designed for visual datasets. While the raw 6.65 TB of per-case field data is not yet publicly available, the demo notebook provides a framework for ingestion, poised for when the data is released post-peer review.

BabyROS is a lightweight alternative to the Robot Operating System (ROS) built on the Zenoh protocol, designed to facilitate simple communication between processes without the complexities of a full ROS installation. It allows data to be published and subscribed to on topics with minimal setup, omitting message semantics to let users define their own schemas. FiftyOne complements BabyROS by transforming the raw data it transports into a structured dataset that can be queried, evaluated, and visualized, enabling real-time curation and error detection in robot perception pipelines. The integration of BabyROS and FiftyOne allows for seamless handling of both 2D and 3D data, supporting live updates and evaluations, and offers a comprehensive toolkit for prototyping and developing AI-driven robotics applications while maintaining flexibility and ease of use.

TwelveLabs and FiftyOne have integrated their technologies to enhance video data handling through natural-language video search, transforming how users interact with video datasets. TwelveLabs provides video foundation models like Marengo and Pegasus, which enable embedding video clips into a shared space with text for easy searching by description, and generating natural-language captions and answers for video content. This integration allows users to embed, search, and caption videos using plain English without the need for local GPUs, as computations occur server-side via the TwelveLabs API. FiftyOne, an open-source toolkit, complements this by offering a visual app for exploring images and video, employing a query language for data slicing, and a "Brain" layer for advanced analysis, thereby facilitating better data curation. This partnership unlocks enhanced functionalities such as natural-language video search, zero-shot captioning, similarity detection, and embedding visualizations, enabling users to manage video datasets more efficiently and intuitively.

TreeMatch, an optimal-transport method for estimating tree density from satellite imagery, is explored through FiftyOne, an open-source tool that provides a deeper understanding of model performance beyond benchmark tables. TreeMatch addresses the challenge of using both strong and weak point annotations, accommodating expert-labeled data alongside noisier pseudolabels, and is benchmarked on the TinyTrees dataset, which includes approximately 11.7 million point-annotated trees across three sensors from Rwanda, China, and France. FiftyOne enhances the evaluation by offering visualization features, such as Keypoints for point annotations, Heatmap overlays for predicted density maps, and sortable fields for per-tile count errors, allowing users to inspect where the model fails and understand label noise. The TinyTrees dataset is available under a CC BY-NC 4.0 license for research and education only, and FiftyOne allows the exploration of TreeMatch using official data loaders and pretrained checkpoints without modifying the code.