What CI/CD strategies work for embedded or IoT projects that require hardware testing?

Embedded and IoT teams encounter unique challenges with CI/CD processes compared to traditional SaaS teams due to their reliance on physical hardware and constrained environments, making it difficult to run tests purely in the cloud. Common issues discussed in forums include managing hardware dependencies, avoiding flaky tests caused by devices, and deciding whether to use existing CI/CD tools or develop custom solutions. To address these challenges, teams often adopt a hybrid pipeline strategy that separates tests into layers, minimizing hardware use and employing simulations and unit tests to cover most scenarios. The approach typically involves using fast feedback layers without hardware and hardware-in-the-loop testing for more critical checks, utilizing device farms and self-hosted agents for orchestrating tests on actual hardware. Additionally, strategies to handle flaky tests, optimize costs, and scale testing include automating device management, implementing retry logic, and parallelizing tests across multiple devices. While some teams consider building custom systems for specific needs, modern CI/CD platforms like Semaphore can effectively manage orchestration, allowing customization for hardware layers without reinventing core CI/CD functionalities.