August 2021 Summaries

Amazon Rekognition is a computer vision API offered by Amazon that allows users to analyze the content of images by detecting and classifying common objects such as furniture and plants. It provides an easy-to-use solution for incorporating image recognition into applications by returning classifications for known items. However, for identifying objects that are not included in Amazon Rekognition's standard model, users need to develop custom models. Roboflow facilitates this process by enabling users to bring their own datasets, label them, train a model, and set up an API for specific classification and object detection tasks. This approach allows for continuous model improvement over time. The concept is part of a broader effort to democratize computer vision technology and make it accessible across various industries.

Brad Dwyer, co-founder and CTO of Roboflow, shares a live-coding session where he develops a computer-vision powered Blackjack basic strategy web app using a playing cards object detection model from Roboflow Universe. The process, which takes just two hours, is streamed online and can be followed by viewers who wish to build the app themselves using sample code. Dwyer encourages viewers to subscribe to their YouTube channel for more immersive content and to access the complete code on GitHub. He invites feedback and suggestions for future live streams, and provides a citation for those wanting to reference the post in their research.

Roboflow has introduced a new capability that allows users to track objects frame by frame in videos and camera streams using their Inference API and a zero-shot object tracking repository, without the need for a separate classifier. Traditionally, object tracking required a separate featurizer model fine-tuned with additional annotations, but this process has been simplified by leveraging OpenAI's CLIP model, which provides robust image features across various domains. By using generalized CLIP object features, users can avoid the additional steps of creating object track annotations and training another model. The process involves constructing a custom object detection model in Roboflow, obtaining an API endpoint and key, and using the zero-shot object tracking repository to track objects without further modeling requirements. This development allows for efficient real-time object tracking on the edge, with the option to improve track performance by increasing the processing frame rate.

Apple's NeuralHash, a perceptual hashing model designed for detecting child sexual abuse material (CSAM), has raised concerns due to potential false positives and vulnerabilities to adversarial attacks. The algorithm generates a 96-bit hash for images, which should ideally match only when images are nearly identical. However, testing revealed instances of distinct images sharing the same hash, both naturally and artificially, highlighting the limitations of a 96-bit space that can inevitably lead to collisions. Despite Apple's claim of a robust system with a low false-positive rate, researchers have found real-world examples of collisions, suggesting that the rate might be slightly higher than reported. Additionally, the possibility of artificially creating images that match CSAM hashes raises further security and privacy concerns. The opacity in the process of adding images to the CSAM database also poses potential risks, as it could be exploited by malicious actors for purposes beyond its intended use.

Researchers have identified vulnerabilities in Apple's CSAM NeuralHash algorithm, allowing the creation of artificial images that produce the same hash as real images, potentially overwhelming Apple's human review system with false positives. Despite this, a similar network, OpenAI's CLIP, was able to differentiate between the real and fake images, suggesting that Apple could integrate a secondary network like CLIP to improve the accuracy of its CSAM detection system. Implementing such a dual-network approach could provide a more reliable mechanism for distinguishing between genuine and artificial CSAM content, mitigating the risk of hash collision exploits. While Apple's current system isn't deemed ineffective yet, the incident highlights the importance of continuous improvement and collaboration in developing robust content moderation tools.

CoreML is Apple's machine learning framework designed for on-device inference, emphasizing the importance of small, low-latency, and low-power consumption models that can be efficiently deployed on Apple devices like iPhones. The .mlmodel file format is optimized for Apple's Neural Engine, enhancing speed and reducing power usage. Users can create CoreML models using tools like CreateML or import models from formats such as TensorFlow's .h5 or Python's .pt, providing an accessible entry point for deploying machine learning applications within the Apple ecosystem.

Roboflow has introduced Roboflow Universe, a platform designed to democratize computer vision by providing open-source datasets and pre-trained models for developers, with the aim of removing barriers such as access to high-quality data and deployment options. Launching with fifty open-source projects, the platform offers a suite of advanced features, including annotation, dataset management, and training, free for public projects, enabling developers to focus on model creation rather than data curation. Roboflow Universe bridges the gap between research and engineering, allowing developers to use state-of-the-art computer vision models without extensive machine learning knowledge. It also offers scalable deployment options, including cloud APIs, SDKs, and on-premise configurations, and promotes community collaboration to improve datasets and models over time. The platform supports both public and private users, encouraging contributions to the open data community while providing robust support for students and hobbyists through its free tier.

In August 2021, Roboflow highlighted several updates and developments in their monthly Changelog. The company focused on preparations for a major product launch on August 11, with backend changes to support this, including improvements in their Health Check's heatmap API and the introduction of Offline Mode. Roboflow Train saw enhancements in model weights conversion, while Roboflow Deploy introduced Enterprise deployment options. The Roboflow Annotate tool received UI improvements, and Label Assist was updated to the latest Roboflow.js version. Additionally, Roboflow expanded its team by appointing Rachel McClung as Head of Product Design, released a free course in collaboration with OpenCV AI Kit, launched an online store with discounts on OAK devices, and opened new job positions. The company also engaged in community activities, including blog posts, YouTube videos, live webinars, and speaking engagements, while featuring user projects like phage counting and autonomous plastic collecting submarines.

Transformers, initially introduced in 2017 for natural language processing, have made significant strides in artificial intelligence, particularly in computer vision, by enhancing object detection capabilities. Microsoft's DyHead has achieved state-of-the-art performance using a Transformer backbone, outperforming previous methods on the COCO benchmark. The evolution of Transformers began with their application in NLP, where models like BERT and GPT demonstrated their ability to predict sequences and mask words, which led to their adaptation in vision tasks. Vision Transformers (ViT) and models like CLIP have further advanced the field by integrating text and image processing, resulting in a web-scale semantic understanding. DyHead's research focuses on directing attention to image features for object detection, marking a notable improvement by incorporating Transformer backbones over traditional CNNs. As Transformers continue to transform AI, their application in tasks such as instance segmentation is anticipated to evolve further.

YOLOX is a cutting-edge object detection model from the YOLO family that enhances model accuracy and training speed by eliminating box anchors and decoupling the detection head into separate feature channels. This tutorial demonstrates the process of training YOLOX on a custom dataset using the Roboflow platform, which facilitates data management, annotation, and conversion into various formats. By utilizing pre-trained weights, YOLOX significantly reduces training time, achieving high Average Precision (AP) benchmarks against datasets like Microsoft COCO. The tutorial covers setting up the development environment, downloading and preprocessing data, training, and evaluating the model, and finally running inference on test images, showcasing YOLOX's efficiency and effectiveness for object detection tasks on edge devices.

ONNX (Open Neural Network Exchange) serves as an intermediary machine learning framework that facilitates the conversion between various machine learning frameworks such as TensorFlow, TensorRT, PyTorch, and TFLite. It aims to provide a standardized platform by implementing a wide range of neural network functions to support cross-framework compatibility. However, due to its open-source nature and the rapid pace of advancements in machine learning research, version compatibility issues may arise, necessitating frequent updates or modifications of models to ensure smooth transitions between frameworks.