December 2023 Summaries

Timothy M's blog post, published on December 28, 2023, details the creation of an innovative exercise tracker that utilizes computer vision and IoT technology to automate the counting of exercises like push-ups. The system employs a sensor node with a camera, running Roboflow's Python SDK, to capture and classify exercise poses through an object detection model. These poses are communicated to an MQTT broker, enabling a JavaScript application to tally the exercises in real time and display the data on a web interface. The project involves collecting a dataset, training a model to differentiate between 'push-up' and 'push-down' states, and using the MQTT protocol for data transmission, with the code available on GitHub and the dataset on Roboflow Universe. This system promises applications in personalized fitness training and remote workout monitoring.

NeurIPS 2023, a prominent conference on Neural Information Processing Systems, featured 3,584 papers showcasing advancements in machine learning and artificial intelligence, with a focus on computer vision and multimodality. The event highlighted award-winning research, revealing trends expected to shape the AI landscape in 2024, such as improved multimodal model performance, enhanced interaction through prompting methods, and advancements in visual logic and reasoning. Among the notable works were innovative models like SEEM, which aims to create a universal segmentation interface, and VisionLLM, which treats images as a foreign language for vision-centric tasks. Other significant contributions included improvements in multimodal datasets through image captioning, novel frameworks for zero-shot detection, and enhanced scene understanding techniques. The conference underscored the potential for these developments to enable new use cases and real-world applications, setting the stage for further exploration and innovation in the field of AI.

CogVLM, a large multimodal model, provides the capability to answer questions about both images and text, offering unique applications in various industries, such as enforcing airport safety, monitoring product defects, and performing optical character recognition (OCR). Despite its end-of-life support, the model is notable for being open-source and deployable on personal infrastructure, distinguishing it from other multimodal models like OpenAI's GPT-4 with Vision and Google's Gemini. CogVLM excels in visual question answering, especially in complex scenarios where traditional object detection models struggle, and supports quantization to reduce memory usage, albeit with a slight trade-off in accuracy. Users can deploy CogVLM efficiently using Roboflow Inference, a computer vision inference server, which facilitates running the model with minimal manual setup.

Piotr Skalski outlines the process of deploying a CogVLM Inference Server on Amazon Web Services (AWS), detailing the necessary setup and requirements for successful implementation. CogVLM, an open-source Large Multimodal Model, excels in tasks such as Visual Question Answering, Optical Character Recognition, and Zero-shot Object Detection. Despite its capabilities, support for CogVLM is ending due to dependency conflicts and security vulnerabilities, and users are encouraged to transition to fully-supported Visual Language Models. The guide provides a comprehensive walkthrough on setting up an EC2 instance, ensuring necessary software like CUDA and Docker is installed, and configuring network and storage settings. By using a client script available on GitHub, users can run inference queries through a Gradio app, with recommendations for monitoring server performance using Docker and NVIDIA tools. The post concludes by highlighting CogVLM’s versatility and potential to replace models like GPT-4V, while encouraging further exploration of its deployment through the provided documentation.

Exploring global cuisines offers insight into diverse cultures through their unique dishes, and image classification, a task in computer vision, can enhance this exploration by identifying foods and providing cultural context. By employing a model from the Roboflow Universe and deploying it in a Streamlit application, users can upload images of food to receive information about the dish's name and origin, fostering cultural appreciation and education. This approach not only strengthens community bonds and supports local economies through culinary tourism but also bridges cultural gaps, promoting global unity. The application utilizes a pre-trained food image classification model, enabling users to identify and learn about various Vietnamese and Korean foods, and serves as a testament to the potential of AI in transforming our culinary experiences by improving food quality checks and enriching our understanding of global cuisines.

Andy Portalatin's master's thesis at the University of Puerto Rico at Mayaguez led to the creation of the "Porifera Classifier," a computer vision model designed to detect and classify up to 126 marine sponge species using the YOLOv8 object detection algorithm on the Roboflow platform. Marine sponges play vital ecological roles, but their identification is traditionally challenging due to complex features and limited taxonomic expertise. The Porifera Classifier automates the identification process, supporting ecological research and conservation efforts by providing rapid, accurate analysis crucial for monitoring sponge populations against threats like climate change and pollution. The model's development involved using Roboflow for annotating and augmenting a dataset of 8,958 images, which was expanded to 27,188 images through various augmentations. Experimental results demonstrated improved model performance, with applications extending to environmental monitoring, citizen science initiatives, and integration with bio-inspired robotics. This innovative approach allows for deeper insights into marine sponge biodiversity and conservation strategies while fostering sustainable practices in marine science research.

The process of detecting brand logos in videos using computer vision involves training an object detection model to identify logos in images or frames of a video, which can help brands and advertisers quantify reach and verify sponsorship requirements. The guide outlines steps to train a custom model or use pre-trained models from platforms like Roboflow, which provides access to a wide range of datasets and models. It details the process of collecting and annotating logo images, training the model, and deploying it using the Roboflow Video Inference API to analyze video frames and plot predictions with Python. This allows users to track the presence of logos in videos, offering valuable insights for businesses in terms of brand visibility and advertising analytics.

CogVLM, an open-source Large Multimodal Model (LMM), is designed to handle tasks involving both text and images, such as visual question answering, document OCR, and zero-shot object detection. Despite its strong performance in qualitative testing, the model is now reaching its end-of-life status due to dependency conflicts and security vulnerabilities, with future support being directed towards fully-supported models like Qwen2.5-VL. Users can still deploy CogVLM on their hardware using Roboflow Inference, an open-source inference server, which supports various quantization levels to optimize the model's RAM usage. The guide illustrates deploying CogVLM on a GCP Compute Engine instance with an NVIDIA T4 GPU using 4-bit quantization, showcasing its capabilities by accurately answering questions about a forklift image, though its performance varies based on the image quality and prompt.

EdgeSAM is an advanced image segmentation model based on the Segment Anything (SAM) architecture from Meta AI, designed to identify object locations in images with increased speed but slightly reduced accuracy compared to its predecessor. Enhanced by the combination with Grounding DINO in the Grounded EdgeSAM setup, this model enables zero-shot object detection and segmentation by generating pixel-level masks for identified objects, making it ideal for applications like auto-labeling datasets for training fine-tuned models, such as YOLOv8. The guide provides a detailed walkthrough on using Grounded EdgeSAM to detect and label objects in images, emphasizing its practical utility in scenarios requiring high-speed performance, such as on mobile devices. Despite trade-offs in mask precision, the use of Grounded EdgeSAM with tools like Roboflow allows for efficient dataset preparation and model training, highlighting its versatility in computer vision tasks.

In December 2023, Google introduced Gemini, a new Large Multimodal Model (LMM) capable of processing text, images, and audio, with initial text integration into Bard and plans for full multimodality. An API was released for developers to integrate Gemini into applications, and the Roboflow team conducted an analysis comparing Gemini to other LMMs like GPT-4 with Vision, LLaVA, and CogVLM across various computer vision tasks, including Visual Question Answering (VQA), Optical Character Recognition (OCR), document OCR, and object detection. During testing, Gemini successfully handled some tasks, such as identifying the number of coins and recognizing movie scenes, but struggled with others, including precise OCR and object detection coordinates. The model's performance was mixed, with some successes and technical issues during testing. Gemini is part of a growing field of multimodal models, highlighting Google's efforts to advance LMM capabilities in the competitive landscape.

Monitoring plant growth using computer vision offers significant benefits for agriculture, ecology, and scientific research by providing precise plant height measurements, which can optimize crop yield and study environmental effects on plant growth. The blog post by Timothy M outlines a methodology where a camera sensor captures plant images, and a computer vision model processes these images to determine plant height. The process involves calibrating the camera with a reference object of known height to establish a pixel-to-inch conversion factor, which is then applied to the plant's bounding box in the image to calculate its actual height. This approach, using the Roboflow platform, involves collecting a labeled dataset, training an object detection model, and running inference to extract bounding box details, which are used to visualize and annotate the plant height on the images. The project successfully demonstrates how this technology can aid in remote plant monitoring, providing researchers and farmers with accurate data to study growth patterns and optimize agricultural practices.

Roboflow has introduced a new class management page designed to streamline the process of managing class ontologies for computer vision projects, which is crucial for reducing annotation errors and optimizing model performance. This page serves as a central hub within the Roboflow dashboard, enabling users to add, rename, delete classes, and manage annotation colors efficiently. A class ontology defines the various object types a computer vision model can recognize, and precise definition of these classes is essential for accurate object identification and classification. Best practices include labeling the most specific classes possible, ensuring consistency among labelers, including some null examples to avoid overfitting, and providing instances with multiple classes when applicable. Users can easily modify their class structures, such as combining classes or remapping them at the version level, and adjust class colors for better contrast or specific use cases. Properly setting up and managing class ontologies can significantly enhance the effectiveness and scalability of computer vision models, and the new tool allows for ongoing adjustments to maintain optimal model performance.

Gemini, a new Large Multimodal Model (LMM) developed by Google, is designed to interact with and answer questions about data in various formats, including text, images, video, and audio. Announced in December 2023, Gemini is described as Google's most capable AI model, with performance reportedly exceeding state-of-the-art results on numerous academic benchmarks. Available through Google Bard and an API, Gemini is set to expand its multimodal capabilities over time. It offers three versions tailored for different use cases: Ultra, Pro, and Nano, the latter being optimized for mobile devices. While Google's evaluations highlight Gemini's strengths in visual understanding, object detection, and few-shot learning, its capabilities in video understanding and speech recognition are also emphasized. Potential applications in computer vision, such as image classification and anomaly detection, are explored, with Gemini expected to integrate into tools like Roboflow's Maestro for improved task performance. However, fine-tuned models remain preferable for real-time and niche use cases due to their specialized training and capabilities.

Google Cloud Vision offers a suite of APIs for various vision tasks, including object detection and optical character recognition, which can be integrated into applications to provide visual intelligence. This guide explores evaluating Google’s Cloud Vision object detection API against other models, using a custom-trained model in Roboflow Universe for comparison. The evaluation process involves selecting a suitable dataset—such as a subset of the COCO dataset focused on people—and testing the models' performance using the mean average precision (mAP) benchmark. The guide details the steps for setting up Google Cloud Vision for testing, running the API against an evaluation dataset, and comparing its results to those of a Roboflow Universe model. The testing reveals that while Google Cloud Vision exhibited improved performance, the results suggest potential similarities in training datasets, like COCO, which may influence the models' accuracy. The guide emphasizes the importance of evaluating models on diverse datasets to determine their suitability for specific use cases, encouraging users to train and test models with real-world data to ensure optimal performance.

YOLOv8, developed by Ultralytics, is a cutting-edge computer vision model designed for object detection, classification, and segmentation tasks. The model, which builds upon its predecessor YOLOv5, can be deployed using the native Python SDK or through Roboflow Inference, a robust and scalable computer vision inference server. This guide explains how to utilize YOLOv8 for object detection, emphasizing the importance of having a trained model, either pre-trained on the Microsoft COCO dataset or customized for specific objects using tools like Roboflow. The guide provides detailed instructions for setting up and running models, from installation to visualization of results, using both Roboflow Inference and the Ultralytics Python SDK. It highlights the capabilities of Roboflow Inference to support extensive computer vision tasks and the utility of the supervision Python package for visualizing and manipulating model predictions, offering users a comprehensive approach to employing YOLOv8 in practical applications.

Brandon Jernigan discusses the transformative impact of computer vision (CV) on data analytics, highlighting its capability to convert visual media into valuable insights that enhance traditional data analysis and machine learning models. By classifying objects, tracking movements, and analyzing time spent in specific zones, CV provides a richer context that traditional data methods cannot achieve, enabling more nuanced inquiries and improved model accuracy. Jernigan outlines practical applications such as historical data analysis for trend detection, supplementing existing data workflows with visual context to improve productivity and sales, and leveraging real-time analytics for immediate decision-making in various sectors like retail and logistics. As technology evolves, the integration of CV into analytics is poised to deepen data-driven decision-making, offering new possibilities for innovation and efficiency.

James Gallagher's article explores the use of the open-source framework Autodistill to automate the labeling of product SKUs for applications such as store checkout systems and interactive museum experiences. The guide demonstrates how to train a vision model using a dataset of vinyl record covers without manual labeling, leveraging tools like Grounding DINO for object detection and CLIP for classification. The process involves downloading a dataset, installing dependencies, and executing a labeling process that uses embeddings to classify images, followed by a review of labeled data using the Roboflow platform. The article concludes with instructions on training an object detection model using the labeled data and testing the model's performance, highlighting the efficiency and speed of using Autodistill for data labeling in computer vision projects.

Roboflow Inference is an open-source inference server that enables the deployment of computer vision models offline on edge devices like NVIDIA Jetson, Raspberry Pi, and other architectures. It allows users to run fine-tuned and foundation models such as YOLOv5, YOLOv7, YOLOv8, CLIP, and SAM for tasks like object detection, classification, and segmentation, without requiring a constant internet connection. The platform is designed to work with Docker, managing dependencies and enabling HTTP requests for model predictions via an API or Python SDK. Users must initially connect to the internet to download model weights, after which models can be run offline, with periodic online updates required. The guide illustrates setting up and running models on images and webcam streams, demonstrating use with the Inference SDK and supervision Python package for visualization.

James Gallagher's blog post, published on December 1, 2023, provides a detailed guide on how to blur people in images and videos using Roboflow's APIs and the supervision Python package to maintain privacy in visual media. The process involves using the Roboflow Hosted API for images and the Roboflow Video Inference API for videos, along with the People Detection model from Roboflow Universe to identify people in the media. For images, pixel coordinates of detected people are used to apply blurring, while in videos, predictions for each frame are retrieved to blur people. The guide includes step-by-step instructions on setting up the necessary Python environment, using the APIs to detect people, and applying blurring techniques, emphasizing the importance of privacy when people are not relevant to a computer vision project.

Computer vision, particularly using the CLIP model developed by OpenAI, offers an automated approach to content moderation in videos, reducing the need for manual human effort. The Roboflow Video Inference API facilitates this process by allowing users to identify specific scenes, such as those containing violence or alcohol, in video content. By comparing video frames to text prompts, CLIP calculates the similarity between frames and predefined categories, helping users implement custom business logic for content management, like restricting certain scenes based on time or audience. The guide provides a step-by-step method to run CLIP on video frames, calculate CLIP vectors, and compare them to moderation labels to determine the presence of specific content, demonstrating its application through Python code. This approach allows organizations to efficiently analyze and moderate video content, deciding whether to flag, review, or restrict based on identified content types.