Medical Data Annotation: Use Disagreement as a Signal

Medical data annotation, particularly in the context of medical imaging, often involves expert disagreement, which is traditionally seen as noise to be averaged out using algorithms like STAPLE. However, in the era of foundation models, such as UNI2 and MedSAM2, where datasets are smaller and more specific, this disagreement should be viewed as a valuable signal rather than a problem. Treating disagreement as a first-class signal can enhance model reliability by identifying edge cases and potential failures. This approach requires explicit representation of disagreements, exploring them through embeddings, and careful curation of datasets to maintain high-quality annotations. Furthermore, regulations like the EU AI Act and FDA frameworks demand comprehensive documentation of annotation quality, making it crucial for teams to adopt workflows that preserve individual annotations and disagreement data. By maintaining detailed records and focusing on disagreement, teams can ensure compliance and improve the performance and reliability of AI models in healthcare settings.