January 2023 Summaries

This article discusses the importance of data curation in computer vision and provides an overview of seven top data curation tools for 2024. These tools are evaluated based on criteria such as annotation support, features, customization, data privacy, data management, data visualization, integration with machine learning pipeline, and customer support. The article aims to help readers find the best data curation tool for their specific use case and budget.

Encord begins 2023 with significant updates to its DICOM tool, aimed at enhancing the annotation and machine learning experience for its partners. Key improvements include a metadata viewer for better annotation context, task management by study for more efficient workflow, and customizable hanging protocols to streamline data viewing. The tool also introduces the ability to annotate in arranged view windows, with plans for further enhancements. Workflow improvements include adjustable benchmark scoring metrics and enhancements to the label editor, such as gamma correction and improved ontology management. Additionally, a new parameter aids in managing large data uploads by skipping duplicates, ensuring smoother operations despite network issues. These updates reflect Encord's commitment to advancing its tools' power and versatility, with further enhancements in workflow management and collaboration anticipated in the coming year.

The text discusses the persistent issue of bias in computer vision datasets, emphasizing the principle of "garbage in, garbage out" in data science. Bias in datasets can lead to skewed outcomes when training machine learning models, with notable examples like Amazon's gender-biased recruitment algorithm and Microsoft's controversial chatbot, Tay. The text identifies various types of bias—such as uneven sample classes, selection bias, and category bias—that may infiltrate datasets through human influence or unintentional dataset simplification. To mitigate these biases, it suggests strategies such as observing class distributions during annotation, ensuring datasets represent the target population, clearly defining annotation processes, establishing quality assurance benchmarks, and regularly assessing model performance. The article highlights the role of Encord, an AI-assisted active learning platform, in reducing bias by providing tools for data annotation, active learning, and model performance analysis, ultimately enhancing the accuracy and fairness of computer vision models.

The text discusses a machine learning engineer's challenge in building an autonomous robot that can collect litter on the ground. The engineer is using a data-centric approach to improve the performance of a Mask RCNN model on the TACO dataset, which contains 60 classes and has issues such as class imbalance, similar object classes, small objects, and low labeling quality. To address these challenges, the engineer uses various strategies including re-labeling bad samples, fixing mislabeled classes, labeling new samples, and data augmentation. The engineer focuses on one class, clear plastic bottle, and improves its performance by 47% from the baseline after two iterations of data labeling and model re-training using the Encord Active tool.

Paul Graham's tweet about Maclean's magazine cover highlighting the "future of the internet" has sparked a discussion on what technology will be used by billions in the future, with AI being a strong contender. Encord believes that AI is at a similar inflection point as the internet was in the 90s and is poised for widespread adoption. Recent advancements in generative AI have made this view more plausible. However, there is an "AI production gap" between proof-of-concept and production models due to issues with AI model robustness, reliability, explainability, caused by a lack of high-quality labels, model edge cases, and cumbersome iteration cycles for model re-training. Encord has released a free open-source active learning toolkit designed to help people building computer vision models improve their data, labels, and model performance.

Computer vision models are becoming increasingly sophisticated and accurate, but their effectiveness relies heavily on the quality of labeled datasets. Poorly labeled or inaccurate data can lead to significant problems for machine learning teams. Common errors include inaccurate labels, mislabeled images, missing labels, unbalanced data, and insufficient data to account for edge cases. To improve dataset quality, organizations should use complex ontological structures for their labels, AI-assisted labeling tools, identify badly labeled data, manage annotators effectively, and utilize platforms like Encord to enhance model development with data-driven insights.

Machine learning engineers need to assess their models' performance, but judging accuracy alone doesn't help identify underlying issues due to the accuracy paradox. Imbalanced datasets can lead to biased models that perform poorly on underrepresented classes. Class imbalance in object detection occurs when properties like foreground-foreground and foreground-background imbalances are not uniformly distributed. Tools like Encord Index address these challenges through data curation, ensuring balanced representation and data quality validation. To solve class imbalance, three major methods exist: hard sampling, soft sampling, and generative methods. Hard sampling involves selecting a subset of labeled bounding boxes to correct the imbalance, while soft sampling adjusts sample contributions and generative methods produce artificial samples to inject into the training dataset. Spatial imbalances also occur in object detection, particularly with regression loss, IoU distribution, and object location, which require specific solutions like stable loss functions, convolutional neural networks, and hierarchical shot detectors. Analyzing data distribution before building an object detection model is crucial to identify and solve these imbalances. Tools like Encord Active visualize outliers and provide metrics for data quality and label-quality analysis, allowing users to build balanced datasets and compare algorithm performance.

The project aims to analyze the Trash Annotations in Context (TACO) dataset, which contains images of litter objects on different backgrounds. The authors use Encord Active, a platform for data analysis and model training, to pre-process the dataset, train a Mask-RCNN model, and evaluate its performance. They find that object area, frame object density, and object count have the highest impact on performance, and most objects are very small. The annotation quality is significantly worse in the unofficial dataset compared to the official one. The model performs well on larger objects but struggles with small undefined objects. The authors use Encord Active to visualize true positives and false positive samples, which helps them identify a class mismatch problem that can be addressed by improving data quality or adding specific post-processing steps to the inference pipeline.

The insurance industry, valued at approximately $6 trillion globally, is increasingly integrating advanced technologies such as machine learning and computer vision to enhance its operations, a trend that is reshaping the sector's landscape. Historically rooted in ancient practices, insurance now utilizes computer vision to improve underwriting accuracy, streamline claims processes, and mitigate fraud, ultimately enhancing customer experience and reducing operational costs. These technologies are pivotal in digitizing documentation through Optical Character Recognition (OCR), facilitating safer claims adjudication via drones, and detecting fraudulent activities using facial recognition and emotion AI, contributing to significant cost savings. Notably, the sector is witnessing substantial efficiency gains, with McKinsey estimating potential revenue increases and cost reductions exceeding $1.1 trillion. As insurers leverage AI to automate underwriting and forward-project risk in industrial insurance, the industry anticipates ongoing innovation in 2023, continuing to drive value for stakeholders and improving the speed and accuracy of services offered.