February 2024 Summaries

Tencent’s AI Lab has developed YOLO-World, a large zero-shot open-vocabulary object detection model that offers impressive performance without the need for data labeling or model training. However, smaller custom models are often more efficient and accurate for specific tasks. This guide explores leveraging YOLO-World's capabilities alongside active learning to train a specialized custom model. The process involves deploying YOLO-World with Roboflow Inference, setting up active learning for data collection, and using the automatically labeled data to train a custom model. The approach allows for the efficient creation of a model that is both fast and accurate, with a reported performance of 98.3% mean average precision (mAP). The guide highlights the ongoing potential to enhance models through continuous data collection and Roboflow's active learning features, ultimately marrying the strengths of large zero-shot models with tailored custom solutions.

Modern computer vision models, such as Contrastive Language Image Pre-training (CLIP) developed by OpenAI, enable sophisticated image search capabilities by using semantic details encoded in image embeddings. This guide demonstrates how to build an image search engine using CLIP and Intel's Gaudi2 system, a hardware optimized for large-scale image processing. By calculating and storing image and text embeddings in a vector database like Faiss, users can efficiently search for images related to specific text prompts or other images. The Gaudi2 system's architecture, featuring 24 Tensor processor cores and substantial memory, significantly speeds up these computations, making it ideal for enterprise applications handling millions of images. The process involves setting up CLIP, computing embeddings, storing them in a vector database, and implementing logic to conduct searches. Such capabilities can enhance media search engines and Retrieval Augmented Generation (RAG) systems, providing rich, semantic search experiences in various applications, from travel exploration to media archive management.

Secure Desk is a real-time computer vision application developed as a visionary project by a new hire at Roboflow, designed to enhance office security by using a vision-powered passcode and alerting system. The application monitors a desk area and sends a push notification if a detected person does not enter a hand-gesture-based passcode within 30 seconds. Utilizing the Roboflow platform, the project involved building an object detection model to recognize people and passcode gestures, supported by data collection through active learning and enhanced with diverse image datasets, including infrared and null images. The application incorporates real-time processing with the Roboflow Inference Pipeline and employs a tracker to manage code sequence entry, ensuring each digit is counted once. The project's implementation and open-source code are available for further exploration on GitHub, showcasing the use of Roboflow's tools in transforming a desk into a secure zone efficiently.

YOLO-World is an advanced zero-shot object detection model that allows users to identify objects in images using arbitrary text prompts without prior fine-tuning. Unlike traditional models with predefined class lists, YOLO-World requires experimentation with various prompts to achieve satisfactory object identification. The model's confidence levels can be significantly lower than those of other models, such as YOLOv8, yet still produce valid predictions. To enhance performance, users can implement strategies like adding null classes, employing two-stage workflows, and utilizing color and size descriptors in their prompts. Additionally, it is suggested to set unique confidence thresholds for different classes to mitigate false positives and negatives. Despite its strengths, YOLO-World faces challenges in spatial prompting and maintaining consistent performance across diverse contexts, prompting recommendations for training custom models for better generalization.

YOLOv9, a new computer vision model architecture introduced by Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao, offers improved object detection performance compared to its predecessors like YOLOv8, YOLOv7, and YOLOv5, particularly when benchmarked against the MS COCO dataset. This model supports object detection tasks exclusively, with no support for segmentation or classification yet. The guide outlines the process of installing and running YOLOv9, including downloading the project repository, training on a custom dataset, and deploying the model using Roboflow's Inference server. It also details the steps for setting up the environment in Google Colab, acquiring model weights, and running inference with both the v9-C and v9-E models. Additionally, the guide provides instructions for training a YOLOv9 model on a custom dataset, such as a football players detection dataset, and highlights the importance of using the right data format and tools like Roboflow for data management and model deployment.

YOLO-NAS is an open-source object detection model developed by Deci AI, leveraging the YOLO architecture and employing Neural Architecture Search (NAS) to optimize its performance. This model, known for its lower latency and higher accuracy compared to predecessors like YOLOv6, YOLOv7, and YOLOv8, can now be trained and deployed on the Roboflow platform. Users can prepare datasets in Roboflow, select the YOLO-NAS training option, and monitor training progress. Once trained, the model can be tested and deployed using Roboflow Inference, an open-source tool that allows for deployment on personal hardware or through the Roboflow hosted API. The guide walks through the process of setting up, training, and deploying a YOLO-NAS model, including the use of InferencePipeline to deploy models on webcams, highlighting the practical application and flexibility of the model in real-time scenarios.

Using computer vision, a juice box quality inspection system can be developed to detect defects such as missing or malformed straws on juice boxes before distribution, ensuring consumers can use the product effectively. The guide outlines a detailed process to create such a system using Roboflow, starting with the creation of a project and uploading images of both acceptable and defective juice boxes. These images are then labeled and used to train a classification model to identify defects. After training, the model can be deployed using Roboflow Inference to operate in real-time on assembly lines, allowing businesses to automatically identify and handle defective products, thus maintaining product quality. The system exemplifies the application of computer vision in industrial inspection, offering a scalable solution for quality assurance in manufacturing.

Apache Kafka can be used to facilitate the integration of computer vision models into existing event streaming infrastructures by sending predictions from models deployed in facilities to a centralized system. This process involves deploying a computer vision model using Roboflow Inference, which can be trained to assess product quality on assembly lines, such as checking the integrity of bottle caps. The guide outlines the steps to broadcast these predictions using Apache Kafka, starting with setting up a Kafka receiver and creating a topic to receive messages. A Python script is used to serialize model predictions and send them to a Kafka consumer, allowing predictions to be visualized and further processed. The approach is particularly useful for applications requiring real-time data processing and centralization of results from multiple models in computer vision deployments.

James Gallagher's guide explores the process of deduplicating image datasets using OpenAI’s Contrastive Language–Image Pre-training (CLIP) model on Intel's Habana Gaudi2 chip, which is optimized for high-performance computer vision tasks. Deduplication is essential for improving the accuracy of multimodal models by removing near-duplicate images that provide limited value, thus reducing model training time and enhancing dataset quality. The guide outlines the steps for installing CLIP, calculating image embeddings, and using Euclidean distance to identify and remove similar images from a dataset, with a focus on a logistics-themed dataset containing nearly 20,000 images. It emphasizes the importance of maintaining the quality of data over mere quantity and suggests using methods like image hashing for exact duplicates while employing CLIP for near duplicates. The guide also discusses the advantages of using vector databases for larger datasets and highlights the iterative nature of model training to adapt to changing requirements and environments, encouraging ongoing refinement through techniques like active learning.

MQTT, a widely supported protocol for data transfer in Internet of Things (IoT) devices and manufacturing execution systems (MES), can be used to broadcast computer vision predictions across a network. The guide demonstrates how to deploy a computer vision model, specifically a bottle cap inspection model, using Roboflow Inference, an open-source server, and broadcast its predictions via MQTT. It involves setting up an MQTT broker and using the Paho MQTT client to publish predictions from the model to the network. The process includes preparing a model with Roboflow, deploying it on a device like a webcam, and using Python code to send predictions to an MQTT broker. The setup allows real-time monitoring and customization of prediction broadcasts, enhancing manufacturing processes by integrating computer vision capabilities into the workflow.

YOLO-World, developed by Tencent's AI Lab, is an innovative real-time, zero-shot object detection model that allows users to identify objects in images through text prompts without prior training or fine-tuning. This model introduces a novel "prompt then detect" paradigm, enhancing speed by eliminating the need for just-in-time text encoding, unlike other zero-shot models such as Grounding DINO. YOLO-World's small version achieves up to 74.1 FPS on a V100 GPU, and it can be deployed on personal hardware using the Roboflow Inference Python package. The guide details the process of setting up and running YOLO-World for object detection, including installing dependencies, importing data, running the model, and visualizing results using the Supervision Python package. Additionally, it addresses challenges such as low confidence levels and overlapping bounding boxes, offering solutions like adjusting confidence thresholds and applying Non-Maximum Suppression (NMS) for improved outputs.

Measuring object dimensions is crucial for pass/fail inspections, especially in high-value or high-volume production settings, and computer vision can automate this process by returning dimensions such as width, length, and height. Two methods for building a dimension inspection system with computer vision are discussed: using a reference object and utilizing depth measurement capabilities. The process involves training an instance segmentation model on Roboflow, labeling images of objects, generating a dataset version, and training a model to detect specific items like ramen boxes and water cases. The reference object approach involves deploying the model to measure dimensions using a known object for conversion calculations, while the depth capabilities approach leverages stereo depth perception technology to calculate dimensions more accurately. Both methods have applications in quality assurance and package sorting systems, offering a logical framework for estimating object dimensions in dynamic environments.

In the context of computer vision and AI, combining multiple domain-specific models for data labeling and training can lead to enhanced accuracy and efficiency, outperforming larger general-purpose models in specific use cases. The guide explores how to leverage the expertise of specialized models, such as person or vehicle detection, to label datasets and train a new model, using tools like Autodistill for automated annotation. The process involves creating or downloading a dataset, labeling it with specialized models, and then training a combined model, which is evaluated against both the specialized models and larger offerings like COCO, Google Cloud Vision, and AWS Rekognition. The results demonstrate that a combined model can achieve higher accuracy and faster inference times than both specialized and large-scale models, while offering the flexibility to be hosted locally or through a managed API, thereby underscoring the practical benefits of integrating domain-specific expertise in model development.

The Roboflow Inference Pipeline offers a solution for deploying computer vision models on edge devices like the NVIDIA Jetson Orin Nano, providing an alternative to the Hosted Inference API that enables real-time streaming applications. This pipeline supports asynchronous interfaces compatible with multiple video sources, optimizing performance specifically for the Jetson's CPU and GPU architectures through tailored drivers and libraries. The guide details the process of setting up the Jetson device using Jetpack 5.1.1, installing necessary libraries for GPU acceleration, and deploying models using Roboflow's inference pipeline and supervision library. It covers how to implement logic for tracking and counting objects in specific zones within video feeds, utilizing tools such as ByteTrack and Supervision to measure metrics like time spent in defined areas. By leveraging the Inference Pipeline, users can enhance real-time computer vision capabilities without compromising on latency, making it suitable for applications like traffic monitoring at intersections.

James Gallagher's guide provides a comprehensive walkthrough on using computer vision to ensure quality control in chocolate box assembly by building an inspection system that verifies the correct number and arrangement of chocolates. The process begins with creating a project on Roboflow, where users can upload and annotate images of chocolate boxes to train a model capable of identifying different chocolate types. The guide details each step, from labeling images to generating dataset versions and training the model, to deploying the system using Roboflow Inference, an open-source server for managing computer vision models. The model's results help ensure compliance with chocolate box specifications by checking that each chocolate is in the correct position and that the box contains the right quantity and variety. The system can be integrated into a broader chocolate quality assurance framework to identify and address defects in individual chocolates before packaging.

James Gallagher's guide, published on the Roboflow Blog, explains how to create a manual assembly quality assurance (QA) system using computer vision to ensure correct parts are selected in the right sequence during manufacturing. The process begins with collecting and labeling images of the parts to be used, such as Lego blocks in different colors, on Roboflow to train a vision model. Once trained, this model, coupled with Python logic, supervises the manual assembly process in real time, displaying warnings if incorrect parts are picked. The system is implemented using packages like Inference for running the model and supervision for tracking and displaying the assembly process. This approach enhances efficiency, reduces defects, and improves product quality by ensuring the correct order of assembly, with potential integrations into broader manufacturing execution systems for better tracking and accountability.

James Gallagher's guide on the Roboflow blog outlines the process of creating a computer vision system for inspecting bottle caps to ensure they are properly sealed before packaging and distribution. The guide walks through setting up a project using Roboflow, gathering and labeling images of bottle caps, generating a dataset, training a model using the Microsoft COCO Checkpoint, and deploying the model with Roboflow Inference. The system is designed to automatically identify sealed, unsealed, and missing caps, allowing for real-time monitoring of defect rates and integration into business processes. The guide also provides resources for expanding the system to other industrial inspection tasks and offers insights into collaborating with AI experts for custom solutions.

Tencent's AI Lab introduced YOLO-World, an innovative real-time, open-vocabulary object detection model that addresses the speed limitations of existing zero-shot models by employing a CNN-based YOLO architecture instead of the slower Transformer-based models. The model, which requires no training, allows users to specify objects through prompts, encoding these into an offline vocabulary to facilitate rapid detection without the need for real-time text encoding. With its "prompt-then-detect" paradigm, YOLO-World significantly reduces computational demands compared to traditional methods, enabling quick and adaptable object detection suitable for real-world applications, particularly on edge devices. It integrates a YOLO detector for feature extraction, a Transformer text encoder, and a Vision-Language Path Aggregation Network for fusing image features with text embeddings, achieving notable performance on the LVIS dataset with impressive frames per second (FPS) outcomes. YOLO-World is 20 times faster and 5 times smaller than other leading zero-shot detectors, paving the way for new use cases such as open-vocabulary video processing and deployment on edge devices without the need for training or data labeling, making it a crucial development in the field of object detection.

The healthcare industry, a significant contributor to the U.S. economy, is increasingly adopting computer vision technology to enhance patient care and operational efficiency. With hospitals facing pressures such as rising costs and labor shortages, computer vision offers solutions like quality assurance for medical equipment, pill identification and counting, automated triage and diagnosis, inventory management, sterile processing, PPE monitoring, and AI-assisted cancer screening. These applications help streamline processes, minimize errors, and reduce waste, thereby improving care quality and lowering expenses. The market for computer vision in healthcare is expected to grow rapidly, reflecting its potential to transform hospital operations by leveraging advanced image and video analysis for various tasks, ultimately supporting medical professionals with more precise and efficient tools.

Introduced in December 2023, Gemini is a series of multimodal models developed by Google and DeepMind, with its latest version, Gemini Advanced, released in February 2024. The Roboflow team conducted qualitative tests to assess its capabilities compared to other multimodal models and the initial Gemini release. While Gemini Advanced showed improvement in certain real-world OCR tasks, such as reading a serial number on a tire, it demonstrated regressions in fundamental tasks like Visual Question Answering (VQA) and Document OCR. The team observed that Gemini Advanced could not process multiple images simultaneously, a feature available in other models like GPT-4 with Vision and Qwen-VL-Plus. Despite these limitations, the model successfully identified anomalies in images and showed promise in some areas. However, the inconsistencies in performance suggest the need for further testing and analysis. The Roboflow team plans to continue exploring Gemini Advanced's capabilities and encourages community contributions to better understand the model's potential.

Computer vision and machine vision are integral to modern manufacturing, offering advanced capabilities to enhance efficiency, quality assurance, and defect detection. While machine vision relies on established rules-based algorithms for tasks like reading barcodes and detecting edges, computer vision leverages artificial intelligence to perform more complex analyses, such as object detection, segmentation, and classification. This technology is used across various industries, from automotive to food production, to identify defects and ensure product quality. Implementing computer vision involves training customized models using business-specific data, which can be integrated into manufacturing processes using devices such as NVIDIA Jetsons or Raspberry Pis. The guide highlights the adaptability of computer vision systems, their potential to improve over time through active learning, and their cost-effectiveness compared to traditional machine vision systems. With the right setup and tools, businesses can significantly reduce defect rates and enhance their assembly lines' stability and quality checks.

James Gallagher's guide provides a detailed walkthrough on training a YOLOv8 Oriented Bounding Box (OBB) model for object detection, particularly focusing on solar panels. Unlike traditional models that produce horizontally-aligned bounding boxes, the YOLOv8 OBB model offers a more precise fit for angled objects by using oriented bounding boxes. The guide begins with data collection from Roboflow Universe, specifically utilizing the Aerial Solar Panels dataset, which initially contains standard bounding box annotations requiring relabeling with polygons for OBB training. It then guides users through setting up a project in Roboflow, annotating images with the polygon tool, creating a dataset version, and exporting data for use in a Google Colab notebook. The training process involves using the ultralytics library to run the model for 100 epochs, followed by testing the model's performance using the supervision package to visualize results. The guide emphasizes the advantages of oriented bounding boxes in providing tighter alignment around objects of interest, demonstrating the improved accuracy compared to traditional bounding box models.

Correct label placement on packages is crucial for efficient shipping and sorting processes, as improperly placed labels can lead to inventory and routing issues. The text provides a guide on using computer vision to verify label placement by leveraging a pre-trained model on Roboflow, which identifies the location of boxes and labels in images. This approach is advantageous as computer vision systems offer flexibility by allowing multiple checks with a single camera, thus reducing costs compared to machine vision systems. The guide details the process of using Roboflow's Universe Model for initial tests and how to deploy the model on personal hardware through Roboflow Inference, using Python packages like supervision for processing outputs. The code provided helps determine whether labels are correctly positioned on packages and highlights the importance of training models with data specific to one's operational environment to optimize performance.

James Gallagher's guide on using computer vision for product counting details a step-by-step process to build a missing item inspection system, particularly useful for ensuring accurate counts in food and drink packaging. The system utilizes computer vision to verify the number of bottles in a package before sealing, employing a model trained to recognize bottle tops from collected and labeled images. The guide walks through creating a project on Roboflow, uploading and labeling images, generating a dataset, and training a model using transfer learning from pre-trained weights. Once trained, the model can be deployed using Roboflow Inference to run on any camera connected to a computer, enabling real-time detection and reporting errors if the detected count is incorrect. The system integrates into business logic to alert and rectify discrepancies in packaging, ensuring quality control in manufacturing and logistics.

Computer vision can be effectively utilized to identify solar panels in aerial imagery, offering various applications such as government analysis for solar panel distribution and insurance verification. This guide demonstrates the process using a pre-trained model available on Roboflow Universe, which achieves a 70% mean average precision score and can be tested or deployed via the Roboflow API or on personal hardware. Users have the option to enhance the model with their data and integrate additional functionalities, like identifying roofs for more precise solar panel counting. The guide also illustrates how to visualize model predictions using the Python package, Supervision, and suggests potential applications for the model, such as monitoring solar panel trends in urban areas.

James Gallagher's guide, "Glass Inspection with Computer Vision," outlines the process of using computer vision models to identify defects in glass with high accuracy, such as scratches, blemishes, and chips. The guide details how to build a vision system by first creating a Roboflow project to store data and models, then collecting and annotating images of glass defects, training a model with this data, and finally deploying the model using Roboflow Inference. The process emphasizes using representative data from the specific environment where the model will be implemented to optimize performance. Gallagher provides step-by-step instructions, including setting up and labeling images with Roboflow, generating a dataset version, training the model from a Microsoft COCO checkpoint, and deploying it for practical use, while also offering guidance on improving model accuracy through preprocessing and data augmentation. This system can be applied to various types of glass, from solar panels to bottles, and the Roboflow sales team is available for assistance in developing custom computer vision solutions for different industries.

James Gallagher's guide explains how to use the Roboflow product recognition API to identify and locate products or detect empty spaces on retail shelves, which can enhance inventory management and ensure compliance with planograms. The API can be used on cloud or personal hardware and requires a Roboflow account to access an API key. By running the Roboflow Inference system, users can deploy a pre-trained model to analyze images and identify products or gaps on shelves, with blue boxes indicating products and yellow boxes showing empty spots. The guide further discusses integrating this product recognition system with a zero-shot classification model like CLIP to match products against a planogram, ensuring they are shelved correctly. This combination allows businesses to automate shelf validation processes, comparing current shelf stock with reference images and prompting human checks if discrepancies are found.

Image classification, a technique used to assign labels to images, has various applications in industries such as manufacturing, logistics, transportation, and retail. This guide details the use of the Roboflow image classification API, powered by OpenAI's CLIP model, which allows users to assign arbitrary categories to images without prior training. By creating a Roboflow account, users can access the API to classify images based on provided tags, and the guide illustrates this process through an example of classifying a scratched car door. For more specific classification needs, the guide suggests training a custom model on Roboflow with user-supplied data and taxonomy, which can result in higher accuracy. The guide concludes with a brief overview of the steps needed to train and deploy a custom classification model, emphasizing the flexibility and utility of the Roboflow API for various classification tasks.

James Gallagher's guide discusses the use of computer vision for wood surface inspection, emphasizing its ability to identify various defects such as dead knots, live knots, and cracks. The guide outlines the process of deploying a pre-trained model using Roboflow, starting with creating an account to access the wood detection model, testing it, and deploying it through the Inference tool on various hardware platforms. It also highlights the importance of customizing the model for specific use cases by collecting and labeling images from one's facility to train a more accurate model. The guide concludes by encouraging users to develop their own wood surface inspection systems for enhanced quality assurance in manufacturing, offering support through Roboflow's sales team for custom solutions.

Automated image captioning can achieve human-level quality through computer vision, and this guide details how to deploy an image captioning API using CogVLM, a multimodal language model, and Roboflow Inference on personal infrastructure. The process involves setting up an Inference server with Docker, using an NVIDIA GPU like the T4, and following a step-by-step approach to install Roboflow Inference, start the server, and generate image captions programmatically. The guide emphasizes the need for a free Roboflow account and provides instructions for using Python code to prompt the CogVLM model to generate relevant captions. It highlights the scalability of CogVLM, which requires significant computational resources, and suggests cloud services like AWS or Google Cloud for deployment, illustrating the process with an example of generating captions for warehouse images.

James Gallagher's guide, published on Roboflow's blog, outlines the process of building an appearance inspection system using computer vision, specifically for identifying visual defects such as scratches on metal. The guide details a step-by-step process that includes creating a project in Roboflow, collecting and labeling data, training a model, and deploying it using Roboflow Inference. Users begin by setting up a project in Roboflow, collecting representative data from their manufacturing facilities, and labeling this data to train a model using bounding boxes. After generating a dataset version and training the model with Roboflow, the guide explains how to deploy the model and integrate it with business logic to manage defective products or alert managers. This comprehensive approach allows users to adapt the system for various types of visual defects across different materials.

Optical Character Recognition (OCR) is a technology that reads text from images or camera feeds and is widely used in modern manufacturing for tasks such as barcode reading, sell-by date verification, and label verification. OCR systems, like the AI-powered DocTR or custom models tailored to specific typefaces, help automate processes such as scanning return slips and verifying product information, thereby integrating smoothly with manufacturing execution systems. While ready-to-use OCR models like DocTR provide quick deployment, they may be slower or less accurate than custom-trained models, which, though more time-consuming to develop, offer higher precision for specific use cases. Training a custom OCR model involves annotating images to build a system that accurately identifies characters, ensuring efficient processing of text data in manufacturing environments.

James Gallagher's guide, published on the Roboflow blog, outlines the process of creating a package inspection system using computer vision, specifically for counting objects like screws in a package. The guide details the steps to train a custom model using the Roboflow platform, beginning with creating a project, uploading and labeling image data, and then generating a dataset. It explains how to train the model, test its accuracy, and deploy it using Roboflow Inference, with an example focusing on counting screws in an image. The system's flexibility allows for the identification and counting of various objects, such as nuts and bolts, and can be adapted to changing processes and products without altering hardware. This approach leverages computer vision's ability to run multiple checks and improve over time with additional data, offering a robust solution for manufacturing and logistics.