How to Train YOLO11 Instance Segmentation on a Custom Dataset

RF-DETR Segmentation, released in October 2025, is a new state-of-the-art model that surpasses the performance of YOLO11 in instance segmentation by being three times faster and more accurate when tested on the Microsoft COCO Segmentation benchmark. The guide provides a detailed walkthrough for training and fine-tuning YOLOv11 on custom datasets, emphasizing its evolved capabilities in both detection and segmentation tasks. It outlines the necessary steps for setting up the environment, using Ultralytics' CLI and SDK for model training and inference, and integrating with Roboflow for dataset management and model deployment. The process includes leveraging a T4 GPU for processing power, utilizing a Roboflow API Key for dataset access, and employing various tools to visualize training metrics, segmentation masks, and bounding boxes. The guide concludes with instructions on deploying the trained model, highlighting its applicability in generating precise pixel-level masks and its adaptability through fine-tuning for detecting custom classes beyond the base set.