Recent advancements in computer vision have sparked interest in their application to medical imaging, offering significant potential to enhance patient care by expediting processes like disease screening and supporting complex diagnostic tasks. However, transitioning from prototype to production in medical imaging systems poses substantial challenges, particularly concerning model robustness in variable practical conditions. Despite strong performance metrics, models often exhibit vulnerabilities when faced with real-world factors such as patient movement and diverse equipment setups, which necessitate comprehensive machine learning testing and robustness analysis. A case study by the Lakera team highlights these issues, revealing that even state-of-the-art models experience critical failures under certain conditions, underscoring the need for thorough testing and improvement before deployment in clinical settings.