How We Use Smoke Tests to Gain Confidence in Our Code

Smoke testing, also known as confidence testing, is a preliminary testing method used to identify severe software failures by focusing on critical software aspects. Originating from electronic hardware testing, the term reflects the idea of checking whether "smoke" emerges when "power" is applied. At Honeycomb, smoke tests are implemented for OpenTelemetry distributions by running example applications in a container and analyzing the telemetry exported to an OpenTelemetry Collector using Bash scripts, jq, and the bats-core framework. This process involves configuring SDKs, employing instrumentation libraries, exporting telemetry, and validating the json output against expected results, such as span names and attributes. The methodology offers several advantages, including demonstrating library usage, ensuring error-free builds in clean environments, and confirming successful telemetry exporting and parsing. By saving telemetry data as json files, Honeycomb achieves repeatable and precise testing across projects using a language-agnostic framework, with results uploaded as artifacts in CI workflows. While smoke tests form a crucial part of the testing suite, they are most effective when combined with unit tests and automated checks, providing confidence throughout a project's lifecycle.