December 2023 Summaries

Datadog's pricing for synthetic browser checks is not as affordable as it initially seems, with costs quickly adding up when considering real-world requirements such as monitoring multiple routes and checking from different regions. The cost can escalate to over $12,000 per month for a basic API setup, making it unaffordable for many startups. In contrast, Checkly's pricing is roughly half that of Datadog, offering a more cost-effective solution with comprehensive testing strategies and tools such as Playwright support and capture trace files. By embracing synthetic monitoring and implementing effective testing strategies, teams can enhance their observability and deliver a better experience to users.

At Checkly, the company prioritizes reliability and uses its own product for monitoring, employing a dogfooding approach to ensure robustness and effectiveness. Shipping daily updates with diverse customer bases presents challenges, including managing complexity in checks that support multiple platforms and regions. To address this, Checkly has implemented a "Smoke Test Matrix" - a collection of checks continuously monitoring the health of its platform, covering various check types and variants. The matrix includes passing and failing checks, as well as alerts for failure, recovery, and non-run scenarios, allowing the company to detect issues and investigate promptly. Leveraging automation tools, Checkly has created a safety net that catches bugs before they impact customers, forming a crucial part of its commitment to high reliability and observability.

Checkly uses multiple tools to monitor their platform and competitors' tools, but still heavily dogfoods their own product due to engineers liking it and finding it easy to monitor. They use Checkly for API Monitoring, Security Monitoring, and Upstream Provider Monitoring, which helps ensure the reliability and speed of APIs globally, security practices, and preemptive issue resolution with upstream providers. This approach showcases Checkly's practical applicability and resilience, reinforcing its position as a robust and adaptive monitoring tool by integrating it into critical facets of their operations.

The key to building reliable applications is a combination of testing and monitoring. Testing ensures that each pull request works before it's merged and deployed to production, while monitoring detects issues that may arise in production, such as database rollovers or third-party outages. By separating testing and monitoring, developers can catch issues early on and avoid surprises for their on-call engineers. The Checkly CLI allows developers to reuse synthetic monitoring checks as an extra layer of tests, providing an additional layer of protection against application failures. This approach helps identify issues in both the application code and the monitoring setup, enabling a more comprehensive testing and monitoring strategy.

To set up an effective observability pipeline for a local Clickhouse instance, one must first have admin access to Prometheus and Grafana. If not already available, these tools can be spun up locally using Docker, with various guides and templates available online. With both tools up and running, the next step is to ensure the Clickhouse instance has the Prometheus exporter enabled, which can be achieved by forwarding port 9363 out of the container. Two primary data sources are required to immediately use some of the fantastic community dashboards: the Prometheus Exporter, which comes built-in to most Clickhouse containers, and the Altinity Clickhouse Datasource, which must be installed manually using the grafana-cli application. Once these data sources are installed and configured, one can begin adding Dashboards to Grafana, importing various community dashboards that require either of these data sources.

This is the ninth part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience. Checkly's new multi-step API check brings together the affordability and functionality of its core checks, offering a cost-effective solution for frequent users, enhanced functionality with dynamic value storage, the ability to work with binary files, and a fully programmable interface. This new feature is a giant leap forward from Checkly's pre-existing API checks, providing users with more versatility and adaptability in their usage scenarios.

Checkly's engineers share practical monitoring tips from their experience in the context of service-level agreements (SLAs) and availability guarantees with large customers. To prove uptime, teams must ensure their services are available for the agreed-upon time, which requires frequent SLA reports and careful frequency selection for monitoring. The choice of frequency significantly impacts the measurable availability, with higher frequencies allowing for more accurate detection of issues and compliance with SLA requirements, such as four nines (99.99%) or five nines (99.999%) availability. By selecting the right frequency, teams can ensure their services meet customer expectations and maintain a high level of uptime.

We implement monitoring checks for scheduled jobs using Checkly's Heartbeat feature to ensure prompt handling of failed job runs and maintain reliable systems with up-to-date data. Our group of jobs includes Infrastructure Jobs, such as sending emails and database cleanup tasks, and Internal Jobs, like data lake updates and feature performance metrics. We utilize Monitoring as Code to create checks for our jobs using the Checkly CLI, deploying them via pinging URLs to trigger notifications when job runs fail. This allows us to catch issues before they affect users, empowering teams to make data-driven decisions with accurate information. By implementing Heartbeat checks, we ensure our systems run smoothly and make timely interventions to address any failed job runs, thereby maintaining the integrity of our application's functionality.

This is the fifth part of our 12-day Advent of Monitoring series. In this series, Checkly's engineers will share practical monitoring tips from their own experience.` The challenge of simulating user experiences in testing while avoiding external and unpredictable elements such as third-party modules and APIs is a significant headache for developers. To tackle this issue, the team adopts a user-centric approach by faking or mocking API calls and libraries for unit tests, and using route interception to bypass loading of marketing tools during end-to-end testing. By preemptively mocking all potential external request failures, they avoid potential issues and achieve a spotless record in their monitoring checks. However, there is still some upkeep required to keep the regex expressions up-to-date, and an issue with report clutter remains to be addressed. The team's experience highlights the importance of using modern testing tools like Playwright and its versatile mocking capabilities.

This article discusses managing artifacts created during end-to-end testing, specifically in the context of Checkly's monitoring platform. The authors aim to keep not only the testing flow but also the maintenance flow contained within Checkly as much as possible. They introduce a method for declaring an API module factory that returns a wrapped API module with two extra methods: `addToCleanupQueue` and `cleanup`. The `addToCleanupQueue` method adds IDs to an internal pile of pending-to-remove artifact IDs, while the `cleanup` method invokes sequentially the cleanup method declared in the module call. The authors also declare an API fixture for showcasing purposes, which is configurable to pass any HTTP client. The fixture includes a global method for iterating over declared modules and invoking cleanup on every one of them. Finally, the article shows how to use this API fixture in checks by adding created entity IDs to the cleanup pile. The goal is to reduce bloat required for garbage collection at the end of each check to a single line of code.

In the context of running self-hosted services or side projects, synthetic monitoring is a straightforward way to keep an eye on things without getting too complicated.` Checkly offers a feature called multi-step API checks that allows developers to implement arbitrary numbers of requests in steps and custom logic to monitor any HTTP-based service. This feature is ideal for advanced monitoring needs without getting too complicated. To set up monitoring for a self-hosted ClickHouse database, developers can use Checkly's `MultiStepCheck` feature by creating four files: `clickhouse.check.ts`, `clickhouse.spec.ts`, `clickhouse-disk.spec.ts`, and `utils.js`. The `clickhouse.spec.ts` file contains basic checks such as pinging the ClickHouse health endpoint and running a SELECT query to check the database version. The `clickhouse-disk.spec.ts` file is used to check if there are at least 5GB of free disk space left. Once the files are ready, developers can deploy them to Checkly using the `npx checkly deploy` command. With this setup, ClickHouse databases can be continuously monitored for issues and receive alerts through various channels such as SMS, phone calls, Slack, or PagerDuty.

We encountered a tricky issue with our public dashboards, which experienced sporadic outages about once every two days. To simulate this scenario, we used ChatGPT4 to generate a simple nodeJS server that would return a 500 status code with a certain probability. We then configured API checks with Checkly, running one every 5 minutes and another every 10 seconds, to detect these outages. The high-frequency API check proved to be more effective in identifying the issue, revealing almost exactly when it started, stopped, and how long it took. This experience highlighted the importance of synthetic monitoring with higher frequency checks for issues that occur sporadically, allowing us to root cause and fix the dashboard's outages and enhance the reliability of our service.

This is the first part of our 12-day Advent of Monitoring series.` is not relevant to the text's main topic, synthetic monitoring. `Hey there! Here is my take on what synthetic monitoring means and why it’s awesome!` sets a friendly tone for the explanation that follows. Synthetic monitoring promotes e2e tests, which simulate real user behavior, by continuously running them against production services. This provides an early warning system, helping developers identify issues before users report problems. It answers questions about feature functionality, auth provider outages, and regional blocking, giving clear signals compared to noisy error logs or metrics dashboards. By leveraging synthetic monitoring with a modern tool like Checkly, developers can keep their apps top-notch and user-friendly.

Checkly has been named a 2023 Winter Intellyx Digital Innovator, reflecting its Monitoring as Code (MaC) workflow and commitment to delivering solutions that help cloud-native engineers solve uptime and reliability challenges with confidence. Checkly is leading the MaC movement by providing a deeply integrated monitoring solution that allows developers to code, test, and deploy their monitoring setups seamlessly. The company's efforts have been acknowledged by industry leaders, including being named in the Gartner Cool Vendor for Monitoring and Observability and featured in two Gartner Hype Cycles. Checkly will continue to innovate in 2024, integrating its solutions into evolving workflows of developers to drive efficiency and reliability at scale.