Auto repair for Together GPU Clusters: from GPU fault to healthy node

Together’s automatic node-remediation system for GPU clusters addresses hardware, host, and network faults through continuous health checks on both virtual machines and underlying hosts, using evidence rather than a generic Kubernetes NotReady signal to determine appropriate actions. Detected issues such as GPU Xid errors, ECC failures, PCIe faults, filesystem problems, InfiniBand instability, and scheduler-reported node failures create auditable repair recommendations, while customers control whether repairs require approval, run automatically, or wait for workloads to become idle. Repairs follow an escalation ladder from VM reboot to reprovisioning, migration to new hardware, and human-initiated removal, with repeat-failure tracking and migration restricted by critical-alert evidence. Organizations can also submit repairs through APIs, the console, or node-label annotations from their own detectors, subject to the same idempotency, capacity, evidence, and safety controls. Safeguards including persistence thresholds, cooldowns, concurrency budgets, circuit breakers, and automatic resolution prevent correlated outages or faulty detectors from triggering widespread remediation. Machines vacated through migration or removal are automatically revalidated with GPU, networking, storage, and NCCL tests before returning to service or receiving a hardware ticket, while the platform is designed to extend beyond any one accelerator vendor.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.