Ground Truth Audit: Wildfire Smoke Detection with FiftyOne

The article explores the importance of conducting ground truth audits on datasets, using the Pyro-SDIS wildfire smoke detection dataset as a case study. It highlights that every object detection benchmark relies on the assumption that human-created ground truth annotations are correct, an assumption often untested but crucial for model quality. The audit, performed using FiftyOne, a dataset curation and evaluation platform, revealed that while the Pyro-SDIS dataset is geometrically clean with no severe train/validation leakage, it suffers from fixed-camera redundancy. The study emphasizes the need for deduplication and camera-aware splits to ensure accurate evaluations rather than splitting frames randomly. It also suggests prioritizing human review on annotation errors supported by multiple independent methods, as single-method findings may be misleading. The article underscores the necessity of triangulating findings across different methods and models to ensure credibility in dataset audits, advocating for a non-destructive, reproducible audit process that adapts to domain-specific challenges like differentiating smoke from fog or clouds.