January 2024 Summaries

In 2023, OpenTelemetry gained significant attention in the observability space, achieving milestones such as going GA at KubeCon, highlighting its vendor-neutral approach and strong market demand. Grafana Labs has been deeply involved with OpenTelemetry, focusing on interoperability with Prometheus and advancing instrumentation SDKs, while also supporting community-driven projects like vendor-specific receivers for the OpenTelemetry Collector. Key developments included integrating OTLP support into Prometheus and enhancing compatibility with Grafana services, as well as launching innovative projects like Grafana Beyla for auto-instrumentation. Looking ahead to 2024, the OpenTelemetry community aims to enhance stability, advance client instrumentation, and explore areas like sustainability and generative AI, while Grafana Labs plans to innovate further in Application Observability and distributed tracing. Efforts to increase diversity within the OpenTelemetry community continue, alongside collaboration with the CNCF on environmental sustainability initiatives.

Spike testing is a form of load testing aimed at assessing a system's ability to handle sudden and extreme increases in utilization, such as those experienced during events like ticket sales, product launches, or seasonal promotions. This testing involves rapidly ramping up the load to a high level without a gradual build-up and often with a very brief or non-existent plateau period, followed by a quick ramp-down. Spike tests are crucial for identifying how a system responds to unexpected traffic surges and may necessitate modifications to scripts or system processes that differ from average-load tests. Preparation for spike tests includes focusing on key processes, monitoring backend performance, and conducting multiple test iterations to fine-tune the system for high-demand events. Performance metrics such as pod speeds, recovery times, and the behavior of critical processes during overload are essential for evaluating system resilience. Tools like Grafana k6 facilitate spike testing by providing configurations to simulate these conditions and aid in optimizing systems to ensure they can withstand real-world spikes.

Stress testing is a method used to evaluate the stability and reliability of a system under heavy load conditions, simulating scenarios where the system experiences higher than average traffic such as during process deadlines or peak hours. This type of load testing involves gradually increasing the load to a level significantly higher than normal, often 50% to 100% above average, but can vary depending on the risk scenarios the system may encounter. It is crucial to conduct stress tests only after completing average-load tests to identify any performance issues under typical conditions before subjecting the system to more strenuous testing. Utilizing tools like Grafana k6, the process involves a controlled ramp-up to an increased load, maintaining that level for a period, and then gradually reducing it, aiming to observe how performance degrades and whether the system remains stable. The results from stress tests help determine if the system can handle extended periods of high demand and inform whether further soak testing is necessary to assess long-term performance under sustained load.

As the demand for seamless user experiences on platforms like Google and Amazon rises, effective load testing becomes essential for application success. The article explores the importance of selecting a robust load testing solution that can handle complex user scenarios and continuous performance testing, emphasizing the evaluation of open-source software (OSS), on-premises vendors, and Software as a Service (SaaS). It discusses the challenges and considerations of building an in-house load testing solution, such as scalability, reliability, and total cost of ownership (TCO), while highlighting the benefits of starting with an open-source tool for proof of concept before potentially transitioning to a more robust solution. Grafana Cloud k6 is presented as an enterprise-grade performance testing solution offering scalability, automation, collaboration, and rich insights, making it an attractive option for teams seeking an efficient and collaborative load testing environment.

Smoke testing is a type of load testing that applies minimal load to a system to verify its functionality and gather baseline performance metrics. Typically conducted with a small number of virtual users (VUs) and over a short duration, smoke tests are useful whenever a test script or application code is created or updated. They help identify errors in test scripts and assess system performance under minimal load, serving as a preliminary check before more extensive testing. Using tools like Grafana k6, smoke tests can monitor the availability and performance of production environments. It's recommended to run these tests with 2 to 5 VUs for durations ranging from 30 seconds to 3 minutes, ensuring scripts run error-free before progressing to more comprehensive testing. If any issues are detected, they should be corrected before other test types are conducted. Grafana Cloud offers accessible options for conducting smoke tests, with a free tier available for users.

The comprehensive guide to load testing websites by Grafana Labs emphasizes the importance of simulating real user traffic to enhance reliability and successfully release new code. It distinguishes between frontend and backend performance testing, highlighting the necessity to test both for optimal user experience and system performance. It discusses various testing approaches such as protocol-based, browser-based, and hybrid scripts, and the differences between component and end-to-end testing. The guide underscores the need to tailor load testing strategies to the specific application being tested, recommending best practices for scripting, executing tests, and considering factors like test environment and load generator location. Additionally, it provides insights into the use of tools like k6 for both protocol and browser-based testing, advocating for a balanced approach that combines backend infrastructure testing with frontend user experience assessments to prevent performance bottlenecks and ensure comprehensive evaluation.

Soak testing is a variation of load testing that focuses on evaluating a system's performance, availability, and stability over extended periods, differing from average-load tests mainly in duration. It assesses potential performance degradation, resource leaks, and other defects that may only become apparent during prolonged use, by maintaining an average load for several hours or days. Essential considerations for conducting soak tests include configuring the duration significantly longer than other tests, reusing average-load test scripts, and ensuring backend monitoring for system degradation. In the context of Grafana k6, soak tests are similar to average-load tests but feature a prolonged load plateau to identify gradual or sudden changes in performance metrics, with the goal of verifying that system performance and resource utilization remain stable over time. Grafana Cloud offers an accessible platform for conducting these tests, with a free tier available to users.

API load testing is a structured process that begins with small, isolated tests and evolves into comprehensive strategies assessing API performance under various conditions. It involves scripting user flows, setting parameters, and grouping URLs to assert performance and correctness. Load is modeled using virtual users or requests per second, with sleep time added to simulate real-world traffic patterns. Tests are designed to verify application responses and ensure they meet service level objectives (SLOs) through thresholds. Error handling and test modularization are critical to managing large-scale tests, and tools like k6 can be used alongside converters and proxy recorders to automate and expand testing capabilities. Although primarily focused on HTTP APIs, k6 also supports protocols like WebSockets and gRPC, with extensions available for additional protocols, allowing for comprehensive performance testing across various systems.

Breakpoint testing is a load testing method used to identify and understand the limits of a system by gradually increasing the load until failure occurs. This approach helps teams prepare for potential system failures by highlighting weak spots and guiding remediation steps. It is essential for situations where system load is expected to grow, resource consumption is high, or significant changes have been made to the code or infrastructure. Conducting breakpoint tests in elastic cloud environments without disabling elasticity is discouraged, as the environment may self-adjust, skewing results. The test involves a slow ramp-up of load, which can be achieved using tools like Grafana k6, and it typically concludes when system limits are reached, allowing teams to decide whether to accept these limits or to tune the system and retest. This iterative process can help extend the system's capacity over time, ensuring it can handle higher loads effectively.

An average-load test is a type of load testing that evaluates how a system performs under typical conditions by simulating the number of concurrent users and requests per second that reflect average production behaviors. This testing process involves gradually increasing the load to a target average, maintaining it for a significant period, and sometimes incorporating a ramp-down period. The goal is to assess system performance during typical usage, identify potential signs of degradation, and ensure that performance standards are met after any system changes. Using tools like Grafana k6, testers can configure scripts to simulate these conditions, allowing them to analyze system stability and resource consumption during the test. Once systems perform well under average conditions, further tests like stress tests can explore performance under more demanding scenarios.

Grafana Cloud has become a popular choice for companies seeking efficient and cost-effective metrics management solutions, prompting many to migrate from open-source software (OSS) to its fully managed, scalable service. Organizations like SailPoint, Dapper Labs, The Trade Desk, and TeleTracking have experienced significant benefits from this transition, including reduced operational costs, minimized system maintenance, improved query speeds, enhanced reliability, and centralized observability. SailPoint, for instance, reduced metrics volume and costs by 33% using Grafana's Adaptive Metrics, while Dapper Labs eliminated storage management headaches, allowing their SRE team to focus on strategic projects. The Trade Desk improved query completion and reliability, reducing developer frustration, and TeleTracking achieved a 50% cost reduction through Grafana Cloud's centralized stack and Adaptive Metrics. These organizations highlight the advantages of migrating to Grafana Cloud, which supports innovation and scalability while offering a robust solution for managing metrics, logs, and traces.

The Observability Journey Maturity Model, developed by Grafana Labs, serves as a tool to help organizations assess and enhance their observability strategies, which are rapidly evolving due to open-source innovations. By answering a series of questions, organizations can evaluate their observability maturity across three main areas: data access, data analysis, and incident response and prevention, with a focus on nine key dimensions. The model assigns one of three maturity levels—Reactive, Proactive, or Systematic—and provides insights into strengths and areas needing improvement, along with recommendations for enhancing observability practices. For example, a Grafana Cloud user utilized the model to identify gaps in cost allocation practices and took steps to address them with new billing and cost breakdown dashboards. The model is publicly available for free, allowing organizations to measure their progress over time and present findings to stakeholders.

TeleTracking, a healthcare operations platform provider, embarked on a journey to enhance its observability platform using Grafana Cloud, integrating tools like Prometheus and Grafana Loki to gain better visibility and control over their services. Initially, they built an in-house observability solution with open-source tools, but as operations scaled, they found the need to transition back to a managed solution with Grafana Cloud due to the complexities and costs involved. The adoption of Grafana Cloud has significantly decreased their workload and improved cost management, particularly with features like Adaptive Metrics that help reduce spend by optimizing metric verbosity. TeleTracking has streamlined monitoring configurations by standardizing key labels across services, simplifying alerting, and creating a "paved road" for AWS Lambda services to enhance developer efficiency. These efforts have not only saved costs but also allowed the company to reinvest in further capabilities such as Frontend Observability and Grafana Incident, ultimately supporting a more proactive and efficient developer environment.

Grafana Labs has announced that it will maintain and manage the Grafana Operator, an open-source Kubernetes operator designed to manage Grafana instances both within and outside Kubernetes environments. This development elevates the Grafana Operator to an officially supported tool, enhancing its reliability, integration with Grafana Cloud, and alignment with GitOps practices. Users can now manage Grafana configurations alongside application manifests using GitOps methodologies, leveraging tools like ArgoCD for continuous deployment. The Grafana Operator facilitates version-controlled infrastructure through Kubernetes custom resource definitions, offering a seamless way to manage dashboards, data sources, and other resources. Official support from Grafana Labs promises better community collaboration, documentation, and feature development, making the operator an essential component for DevOps engineers, SREs, and infrastructure architects seeking to streamline observability efforts. The operator is easily installed via Helm, and comprehensive guides and documentation are available for those looking to integrate it into their workflows, ultimately supporting a code-centric approach to managing Grafana resources.

Grafana Beyla 1.2 introduces full Kubernetes support, enhancing its auto-instrumentation capabilities by allowing users to decorate metrics and traces with Kubernetes metadata such as pods and deployments. This update eliminates the need for an intermediate OpenTelemetry collector, providing a richer experience when visualizing and querying metrics and traces. The DaemonSet deployment mode is now preferred, and the new service selectors enable precise selection of services to instrument based on Kubernetes attributes like namespaces, deployments, and pod names. This version simplifies deployment and operation by allowing users to specify high-level selection criteria, enhancing the identification and grouping of instrumented applications while maintaining the versatility of the service selectors. By integrating Kubernetes-specific service selection criteria, Beyla 1.2 allows for better application observability and streamlined operations, significantly improving the user experience.

Grafana 10.3 introduces several enhancements, including advanced controls for anonymous access, multi-stack data source configuration, and improved visualizations with features like pan and zoom for canvas panels and enhanced tooltips for better data interpretation. The release also upgrades Grafana Alerting and log analysis capabilities, making monitoring and troubleshooting more efficient. Users can now track metric changes over time in stat panels and visualize system health with time series and state timeline visualizations. The update includes better organization and visibility for alerting contact points, improved log analysis with a new popover menu, and the ability to share complete table data in PDF reports. New features aim to streamline querying processes, enhance security and resource management, and offer a more intuitive user experience across all editions of Grafana. The release of Grafana 10.3.1 resolves technical issues without introducing breaking changes, ensuring a smooth transition for users upgrading from earlier versions.

Grafana Tempo 2.3 introduces a new storage format, vParquet3, which includes dedicated attribute columns to enhance TraceQL query performance. These columns offer significant improvements in query speed and resource utilization by storing high-cardinality attributes separately, thereby reducing memory usage and increasing efficiency. The feature allows users to configure up to 10 string attributes at the resource and span levels, which are then stored in dedicated columns. This configuration should be approached strategically, either by analyzing data patterns or query usage, and is not meant to be frequently changed due to the creation of new shards in the block space. Real-world tests have shown notable reductions in query tail latency and resource consumption, with memory and CPU usage dropping significantly. Future updates may include automated configuration generation to accommodate multiple tenants and ease the setup process.

Grafana Labs, initially known as Raintank and founded in 2014, aimed to create a sustainable business around the open-source Grafana project by using revenue from commercial offerings to reinvest in the technology. Despite early challenges in monetization and roadmap clarity, Grafana Labs experienced success through a community-driven approach and a big tent strategy, making Grafana compatible with over 100 data source plugins. This led to the development of commercial products like the Grafana Cloud platform and Grafana Enterprise offerings. With a focus on technology for practitioners rather than buyers, Grafana Labs now serves over 3,000 customers, including major companies like Bloomberg and Salesforce, and dedicates 80% of its engineering resources to open-source projects. The company's journey is captured in "The Story of Grafana," a documentary series reflecting on its origins and growth into a robust open-source community with over 20 million users.

In an effort to enhance the monitoring of MySQL NDB clusters, the text outlines a method involving Grafana, InfluxDB, Loki, and various scripts and tools to efficiently track the performance and health of the system. The MySQL NDB platform, known for its complexity due to its multiple components including ndb_mgm, ndbd, and MySQL processes, poses significant monitoring challenges. Instead of relying solely on MySQL Enterprise Manager, the text suggests using Grafana for real-time monitoring, supported by the collection of data through scripts that poll MySQL API nodes and store metrics in InfluxDB, while logs are managed with Grafana Loki. The process includes setting up alerting systems using Grafana to notify administrators of potential issues in the cluster. The implementation involves utilizing tools like Telegraf for process monitoring and Promtail for log ingestion, ensuring that errors are detected promptly and that the system operates efficiently. This approach fosters a proactive stance in managing database clusters, offering a more automated and less maintenance-intensive solution than traditional methods.

Grafana Labs has taken over the management of the Zabbix plugin, a popular third-party data source for Grafana, originally developed by Alexander Zobnin in 2015. The plugin allows users to visualize data from the Zabbix monitoring system within Grafana dashboards, enhancing the dashboarding experience for Zabbix users. Zobnin, who was initially motivated by limited dashboarding capabilities in Zabbix, developed the plugin with support from the open-source community and Torkel Ödegaard, Grafana's creator. The plugin quickly gained popularity, reaching around 60,000 active installations. As the sole maintainer for eight years, Zobnin found it increasingly difficult to manage alone, prompting Grafana Labs to take ownership, ensuring the plugin's sustainability and growth. This transition allows Zobnin to focus on future developments and other projects while ensuring the plugin remains free and open-source, benefiting the community and facilitating long-term usage.

Grafana Cloud's Incident Response & Management (IRM) suite, which includes tools like Grafana Alerting, Grafana OnCall, Grafana Incident, and Grafana SLO, provides a comprehensive solution for efficiently handling incidents by streamlining the process of detecting issues, escalating alerts, automating tasks, and deriving actionable insights. Three companies—Clearco, Prezi, and Ultimate—have adopted Grafana Cloud to enhance their incident response strategies, achieving improved communication, reduced costs, and greater efficiency. Clearco transitioned to Grafana IRM to unify their alerting strategy, leading to faster responses and reduced alert fatigue. Prezi switched from PagerDuty to Grafana IRM for cost savings and operational simplicity, finding the transition smooth due to Grafana's feature parity and ease of use. Ultimate integrated Grafana Cloud to unify their observability strategy, resulting in expanded use of dashboards and logs while improving incident response workflows. These companies have not only enhanced their incident management processes but also realized significant cost savings and increased operational efficiency through their adoption of Grafana Cloud.

GrafanaCON 2024, scheduled for April 9-10 in Amsterdam, marks the first in-person gathering of the Grafana open-source community in five years, offering a platform for technical talks, live demos, and community interaction. Although the event is sold out, virtual participation is available through a keynote recap webinar. Attendees will engage with contributors and maintainers of key projects like Grafana, Loki, and Prometheus, learning how to navigate and contribute to open-source initiatives. The conference also invites submissions for presentations that showcase real-world applications and creative uses of Grafana, with a call for papers open until February 20. Additionally, the Golden Grot Awards will recognize outstanding Grafana dashboards, with top entries eligible for a trip to Amsterdam to be honored at the event. This gathering not only promises technical insights and innovations but also emphasizes the value of community connections and shared learning experiences.

Navish Bahl shares his journey of setting up a home automation system using Grafana Cloud and Home Assistant, inspired by his previous manager's use of similar technology. He outlines the necessary hardware, including a Raspberry Pi and various smart devices, and explains how to configure Home Assistant as the central software for managing these devices with a focus on local control and privacy. Bahl details the integration of MQTT to facilitate communication between devices, as well as the installation of Prometheus for data metrics. He emphasizes the benefits of visualizing and monitoring home automation through Grafana Cloud dashboards, which allow for real-time tracking of household statistics such as temperature and humidity. Furthermore, he discusses the utility of Grafana Alerting, which enabled him to receive timely notifications about battery levels in security sensors, ensuring peace of mind during a planned vacation. Bahl encourages readers to embark on their home automation journey, highlighting the ease with which Grafana Cloud can be leveraged for effective monitoring and alerting.

Amazon EC2, a foundational service in the AWS ecosystem, has been pivotal in popularizing cloud computing, but monitoring its diverse range of instances can be challenging. Grafana Cloud introduces a new EC2 monitoring solution, designed for simplicity and efficiency, without the need for managing local agents. This solution, leveraging Grafana Scenes, offers a streamlined experience by focusing on essential metrics, enabling users to view large numbers of instances across accounts and regions, filter based on specific criteria like AWS account ID or region, and quickly identify instances that need attention. Users can drill down to instance-level detail to diagnose issues and monitor resource scaling by region or tag, with an emphasis on optimizing resources and reducing costs. To access this functionality, users need to configure the AWS integration via Grafana Cloud, using either automatic or manual methods for setting up IAM roles. The platform provides out-of-the-box dashboards for a comprehensive overview, promoting a consistent experience across various data sources, and is available on a generous forever-free tier.

Released on December 18, 2023, Grafana 10.2.3 unintentionally included features and two minor breaking changes meant for the upcoming 10.3 version, due to a mistake in the release process. Despite the risk of increased bugs, the release remains available to avoid disrupting existing pipelines and Docker tags, as the features have been tested on Grafana Cloud. Users are advised to review the changelog and approach the upgrade as they would a minor version update, particularly noting changes that impact field transformations and data source permissions. Key new features include enhancements in data visualization, such as improved tooltips and the ability to plot enum values, as well as updates to Grafana Enterprise, like the ability to monitor anonymous access and new dashboards for the Azure Monitor data source. The release also introduces a feature for performing regression analysis through data transformation and updates to Grafana-managed alerts, particularly for Opsgenie integration. For a comprehensive list of changes, users are directed to the changelog and provided guidance for upgrading their setup.

In 2024, the observability sector is expected to see significant advancements, with crucial trends such as AI integration, cost reduction, and the convergence of application and infrastructure observability gaining traction. AI is anticipated to become a more integral tool in observability, aiding in tasks like signal correlation and anomaly detection, thereby freeing up human resources for strategic work without replacing jobs. Concurrently, companies will focus on better budget management to handle economic challenges, with a more data-driven approach in FinOps enabling greater financial observability and informed decision-making. The convergence of application and infrastructure observability, particularly through tools like OpenTelemetry and Prometheus, will facilitate the resolution of performance issues in complex distributed architectures. OpenTelemetry is set to enhance its profiling capabilities, offering deeper insights into application performance, while CI/CD observability will be strengthened by OpenTelemetry standards, ensuring greater reliability and performance efficiency in software delivery pipelines.

Engineers Haris Rozajac and Nathan Marrs at Grafana Labs explored transitioning from individual contributors (ICs) to engineering managers (EMs) to expand their leadership skills and tackle organizational challenges at a hyper-growth startup. They discovered that Grafana Labs supports a dual-track career progression, allowing engineers to switch between IC and EM roles without losing their current standing. Through interviews with colleagues who have successfully made this transition, they learned the importance of self-advocacy, organizational environment, and luck in facilitating a move to management. Key attributes for a successful EM include empathy, effective communication, technical proficiency combined with delegation, and a focus on mentorship. The authors emphasize the importance of being prepared for opportunities, advocating for oneself, and seeking experiences that enhance leadership capabilities. Grafana Labs' supportive environment and insights gained from peers have bolstered their confidence as they pursue this career path.

The guide explores the complexities of handling out-of-order and older logs in Grafana Loki, highlighting the historical challenges and the engineering trade-offs made to address these issues. Initially, Loki's design rejected logs that were older than the most recent entry received, which posed significant inconveniences for users with real-world scenarios. The solution implemented allows Loki to accept delayed logs within a defined timeframe, with a focus on maintaining performance and minimizing operational costs. The guide delves into the technical configurations necessary for log ingestion, explaining the importance of settings such as max_chunk_age and query_ingesters_within, and how they impact the querying and storage of older data. Users are advised to consider the implications of altering these configurations, as doing so could lead to increased costs and decreased performance. The guide emphasizes that while adjustments can be made to accommodate specific needs, users must weigh the potential operational trade-offs.

Antonio Calero Merello's blog post provides a detailed guide on setting up alert rules to monitor sensor data using Grafana and Raspberry Pi, focusing on temperature and humidity levels. The tutorial builds on a previous guide that explains how to use a temperature and humidity sensor with Prometheus for data collection and Grafana for visualization. It explains the process of using Grafana Alerting to establish notifications when sensor data falls outside specified ranges and how to send these alerts to a webhook endpoint. The guide covers the complete setup, including cloning a repository for the project, configuring Grafana, setting up contact points for notifications, and creating alert rules for both temperature and humidity metrics. It also highlights how to configure notifications with summary annotations and provides insights into Grafana Alerting's functionality, encouraging readers to explore further with a suggested follow-up tutorial.

Working as a product designer at Grafana Labs offers a dynamic and challenging experience, particularly due to the technical complexity of engineering tools and the need for strong collaboration across teams. Catherine Gui, a product designer at Grafana, describes her role as creatively engaging, involving user-centric design processes and the necessity of building strong relationships both within and outside the company. Despite the challenges of not having a technical background, she emphasizes the importance of communication and collaboration with engineers and subject matter experts to understand and solve user problems effectively. Gui shares insights on developing domain knowledge and leveraging allies to navigate the complexities of product design, highlighting the value of sharing work frequently and incorporating feedback from experts. Her experiences underscore the significance of teamwork in bridging gaps between UX and engineering, ultimately leading to more confident and successful design outcomes.