March 2022 Summaries

Grafana Labs embarked on a creative project during their company-wide hackathon to explore whether Grafana, traditionally a data visualization tool, could run the classic game Doom. Inspired by a previous project that integrated Rick Astley's music video with Grafana, the team took on the challenge and successfully implemented "Doomfana" by using time series panels and Grafana Live streaming features. The project involved two main approaches: using WebSockets for server-side execution and WebAssembly (WASM) for browser-based operation, with the latter ultimately chosen for better performance. This endeavor not only showcased the versatility of Grafana's live streaming capabilities but also provided insights into optimizing time series panel performance. The team plans further improvements and features for "Doomfana," and the project highlights the innovative spirit fostered during Grafana Labs' hackathons.

Grafana Mimir is introduced as the world's most scalable open-source time series database, designed to manage over one billion metrics with features like high availability, multi-tenancy, and rapid query performance that surpasses its precursor, Cortex, by up to 40 times. Originating from the Cortex project initiated in 2016 to offer a scalable Prometheus-compatible solution, Mimir incorporates advancements from Grafana Enterprise Metrics and Grafana Cloud, all under the AGPLv3 license to balance open-source contributions with commercial benefits. Mimir simplifies deployment and maintenance, addressing previous technical debts, and can serve as a drop-in replacement for Cortex, facilitating a seamless migration. It supports not only Prometheus metrics but is also set to integrate with other formats like Influx and OpenTelemetry, embodying Grafana Labs' "big tent" philosophy by forming a comprehensive observability stack alongside Loki, Grafana, and Tempo.

Grafana Mimir is an open-source, scalable time series database for metrics, developed by Grafana Labs, which combines Cortex and previously commercial code, and is licensed under AGPLv3. Aimed at improving scalability and ease of use, Mimir features a horizontally scalable architecture, supports high cardinality queries, and offers enhanced documentation and configuration compared to Cortex. It can ingest metrics from various sources such as OpenTelemetry and Datadog, while maintaining compatibility with Prometheus. The project is part of Grafana's strategic shift to open-sourcing features from Grafana Cloud Metrics and Enterprise Metrics, aligning with the company's commitment to open-source and sustainable business models. Mimir is designed to be user-friendly and prevent lock-in, allowing users to migrate to other Prometheus-compatible solutions if desired. The launch of Mimir represents a significant step in Grafana's ongoing partnership with major cloud providers and its pursuit of an AGPL-licensed "LGTM" stack, which includes Loki, Grafana, Tempo, and Mimir.

Grafana Labs and New Relic have collaborated to enhance functionality for joint users through the New Relic data source plugin available to Grafana Cloud and Grafana Enterprise users. This plugin simplifies the configuration process and enhances data interaction by allowing users to access New Relic data via a "point-and-click" method or NRQL, New Relic's query language. It supports template variables, enabling dynamic dashboards by suggesting NRQL queries and providing comprehensive views of application data. Additionally, the plugin facilitates data unification, offering the ability to visualize New Relic data alongside other data sources like Prometheus, ServiceNow, and GitHub, thus allowing a holistic view of services and applications. This unified approach ensures users can link alerts to a singular dashboard for detailed analysis and improved observability. For further information, users are encouraged to visit the New Relic solutions page or trial Grafana Cloud Pro for a comprehensive observability experience.

GrafanaCONline 2022, set to begin on June 13, invites participants to join a diverse lineup of over 30 live online talks featuring experts from Grafana Labs and the broader observability community. The event encourages contributions from individuals using Grafana's open-source tools, Grafana Cloud, or Grafana Enterprise, whether for personal, academic, or business purposes. Applicants for a 30-minute speaking slot are asked to share stories of problem-solving and innovation, supported by demonstrations and examples of Grafana dashboards. The call for proposals is open until April 1, with a commitment to highlighting varied and inclusive experiences, including those from first-time speakers. This event aims to foster community engagement and inspiration through unique and impactful use cases of Grafana technology.

RC Thériault is a team member at Grafana Labs who plays a versatile role in SDR enablement, including scheduling, facilitating, and developing sessions. Thrilled to work in a rapidly growing company with familiar colleagues, RC has had a fulfilling six months at Grafana Labs, particularly enjoying a recent enablement roadshow for the GTM organization and delivering improv workshops in person. Previously part of the improv troupe Hulk Spanx in Chicago, RC fondly recalls the camaraderie and creative synergy with the team, which included marrying a fellow troupe member. Now living in Charlotte, North Carolina, RC enjoys traveling, has discovered a passion for basketball, and finds EDM music essential for work focus. A proud Canadian, RC adores poutine and finds comfort in watching "New Girl."

Grafana Cloud has introduced a full trace retention search feature in beta for its Grafana Cloud Traces, extending the capabilities of recent trace searches to a 30-day retention period across all Google Cloud Platform-based clusters in the U.S., Europe, and Australia. This enhancement is built on serverless technologies, allowing rapid data scanning by deploying hundreds of Google Cloud Functions, although the search is limited to a maximum of 24 hours at a time. Currently, the search throughput ranges from 30 GB/s to 60 GB/s, but efforts are underway to further optimize this by potentially adopting new data storage methods like columnar formats. Despite being in beta, the feature aims to solidify Tempo's position as a leading distributed tracing backend, with further innovations planned in metrics generation. Users can engage with the Grafana Labs community for updates and support, while those new to Grafana Cloud are encouraged to explore its offerings, including a free tier for metrics, logs, traces, and dashboards.

Grafana Labs has investigated the potential impact of the Okta Lapsus$ breach and confirmed that they have not been affected. Although Okta confirmed a breach impacting some customers, Grafana does not use Okta for customer authentication unless specifically configured by clients and does not store customer data in Okta. Internally, Okta serves as Grafana's primary Directory and Identity Provider. Grafana has reviewed internal service logs and found no suspicious activity, correlating Okta user activity with external systems. As a precaution, Grafana is enhancing its logging and forensic capabilities, reviewing IdP systems, and considering improvements to its internal authentication systems to reduce reliance on third-party services. Grafana encourages users to report security vulnerabilities via email and provides a PGP key for encrypted communication. Security updates and announcements are regularly posted on their blog, with an option to subscribe to an RSS feed for updates.

Grafana OnCall is a tool introduced with Grafana 8.0, designed to enhance on-call management by integrating with Grafana Cloud to streamline alerting workflows. It centralizes alerting information, allowing users to configure on-call schedules and escalation policies, and customize notifications based on personal preferences. The tutorial demonstrates setting up team member details, importing schedules from Google Calendar, and adjusting notification methods to ensure timely and accurate delivery of alerts. To explore the tool further, users can join a free webinar or try a 14-day trial of Grafana Cloud Pro, which offers comprehensive observability features.

Relabeling in Prometheus is a versatile tool that enables the classification and filtering of targets and metrics by modifying their label sets, which is crucial for managing Prometheus metrics and applies to Grafana Agent users as well. Labels, which are key-value pairs, help in organizing and understanding what's being measured, while relabeling rules can be applied at different stages of a metric's lifecycle, from selecting scrape targets to storing data in Prometheus' database and sending it to remote storage. The process involves configuring relabeling steps in various parts of the Prometheus configuration file, such as scrape_configs, metric_relabel_configs, and write_relabel_configs, each serving different purposes. A relabel_config block comprises several fields, including source_labels, separator, target_label, regex, modulus, replacement, and action, which can perform actions like keep, drop, labelkeep, labeldrop, replace, hashmod, and labelmap to manipulate label sets. These actions can filter targets and metrics, modify label values, scale Prometheus horizontally, and map label pairs to new names, with common use cases including ignoring applications, splitting targets across servers, and managing high-cardinality metrics.

Kelly Pasztor, the Director of Learning and Development at Grafana Labs, plays a crucial role in expanding the organization's learning initiatives as it grows. She is dedicated to fostering an environment where employees, known as Grafanistas, can thrive through various development opportunities, including mentoring, coaching, and leadership training. Kelly joined Grafana Labs to be part of a dynamic and rapidly expanding team, drawn by the company's innovative culture and the chance to make a significant impact. Her notable projects include creating internal "shortcasts" to promote company culture and a self-directed Leadership Development curriculum. Outside of work, she enjoys family activities, traveling, and coaching, and recommends using the Blinkist app for quick book summaries. Kelly's adventurous spirit is evident in her bucket list achievements, such as visiting the Galapagos Islands and engaging in thrilling experiences like bungee jumping and skydiving. For those working from home, she suggests implementing a "virtual commute" to transition from work to personal time.

Vitaly Zhuravlev discusses a new integration for Grafana Cloud, which now supports Gitea—an open-source platform for hosting version control using Git, along with collaborative features such as bug tracking, wikis, and code reviews. This integration allows users to leverage Grafana's observability capabilities to monitor metrics, logs, and traces across their software development projects hosted on Gitea. With the release of Gitea 1.16, new analytical metrics have been introduced, enabling insights into repository usage and performance issues. The Grafana dashboard offers key statistics, such as real-time user activity, repository metrics, and issue tracking, alongside performance indicators like CPU and memory usage. This tool aids in identifying new issues, understanding usage patterns, and assessing team activity over specified periods. Grafana Cloud offers a free tier for users to easily begin utilizing these features, and further resources for monitoring and alerting are available through Grafana's documentation and community channels.

Amon Reich, founder of SmartLiving.Rocks, has developed the SeniorenSmarthome project, which utilizes Grafana dashboards and open-source technology to support Ambient Assisted Living (AAL) for elderly individuals, enabling them to maintain independence in their homes. Motivated by the need to monitor his father's well-being, Reich adapted existing smart home devices to track activity using sensors connected to a Grafana dashboard, providing real-time insights into daily routines and offering peace of mind without the high costs of hiring aides or moving to assisted living. The system, which runs on an Intel NUC with Proxmox and Home Assistant, leverages Grafana's intuitive dashboard creation and features like interval functionality and stacking to present meaningful data visualizations. A strong advocate for open-source collaboration, Reich encourages others to contribute to enhancing AAL solutions by sharing insights and improving algorithms, with the aim of creating accessible and effective support systems for the aging population.

In March, Grafana Labs is hosting a series of free webinars focusing on key topics in observability, including log data privacy with Grafana Loki, scaling distributed tracing with Grafana Tempo, and unified alerting and on-call management in Grafana. The webinars, led by Grafana Labs experts, aim to address challenges such as complex data privacy laws, securing and scaling distributed tracing tools, and improving alert management. Participants can learn about integrating privacy techniques into logs, using Grafana Enterprise Traces for scalable tracing solutions, and leveraging Grafana's enhanced alerting features and the Grafana OnCall tool to improve incident response. These sessions offer live demos, Q&As, and insights into utilizing Grafana's capabilities effectively within enterprise environments.

Grafana is a widely-used platform for observability and data visualization that requires effective organization of teams and resources to optimize security, simplicity, cost attribution, and customization. To achieve this, Grafana offers various organizational structures, such as Folders and Teams, which are recommended for managing access to core resources like dashboards and alerts due to their flexibility and ease of use. Instances or Stacks provide true isolation for teams, ensuring no information leakage between them, while Orgs offer less flexibility and isolation, making them a less favored option. Grafana's organizational tools, such as provisioning and APIs, facilitate seamless resource management and team onboarding, allowing for customization and synchronization with SSO providers. Grafana Labs plans to enhance the capabilities of Folders and Teams, focusing on personalized experiences and improved provisioning, while investing less in Orgs due to their limited flexibility. Overall, Grafana's organizational strategies aim to streamline management and collaboration, ensuring secure and efficient access to data while accommodating diverse team needs.

Grafana Labs, recognized for its unique culture and innovative contributions to enterprise technology, has recently been featured on two prestigious lists: The Enterprise Tech 30 and America's Best Startup Employers. The Enterprise Tech 30, curated by Wing Venture Capital, highlights promising private companies addressing enterprise challenges such as data, collaboration, and security, and Grafana Labs was acknowledged for its observability platform, securing a spot in the late-stage startup category. Additionally, for the second consecutive year, Grafana Labs was named one of America's Best Startup Employers by Forbes and Statista, which evaluated companies based on growth, reputation, and employee satisfaction. This recognition aligns with Grafana Labs' commitment to fostering a transparent, growth-oriented, and creatively autonomous work environment, which attracts and retains employees who resonate with its mission and values. The company's focus on radical transparency and respect for employee empowerment has contributed to its success and industry recognition.

Synthetic Monitoring in Grafana Cloud is a tool designed to simulate user experiences and provide a global perspective on application performance, utilizing probes located in 22 global sites to monitor services. The platform offers the ability to visualize and alert on data, similar to other Prometheus metrics, and includes predefined alerts to simplify the alerting process. Best practices for creating effective Synthetic Monitoring alerts include using recording rules to manage multiple alerts on a single metric, adjusting alert expression ranges to prevent false alerts, and avoiding probe-level alerting due to potential network issues. Users are encouraged to test alert expressions using Grafana Explore to identify potential problems before they occur. Additional resources and support are available for those interested in optimizing their use of Synthetic Monitoring and alerting within Grafana Cloud.

Grafana Labs offers an integration for monitoring Apache Spark clusters using Grafana Cloud, focusing on helping users navigate their observability journey. Apache Spark is a versatile data processing framework known for its ability to quickly handle large datasets by distributing workloads across multiple instances. This integration emphasizes monitoring the Worker Nodes, which are pivotal in executing tasks, through a unified dashboard that provides insights into active jobs, worker count, core usage, and memory consumption. The integration leverages a Prometheus plugin and the Grafana Agent to scrape data from Spark nodes. Future updates aim to include monitoring capabilities for the Driver Program. Users are encouraged to provide feedback through the Grafana Labs Slack Community, and there is a free tier of Grafana Cloud available for new users.

GrafanaCONline 2022, a virtual event hosted by Grafana Labs, is set to launch on June 13 and will feature five days of live talks, demos, and workshops focused on tools like Grafana, Grafana Loki, Prometheus, and Grafana Tempo. The event will highlight new features, including the Grafana 9.0 launch, and it expects over 20,000 participants from the global Grafana community. Open to a diverse range of speakers, the Call for Proposals (CFP) invites submissions for 30-minute talks on various projects, with the deadline set for April 1. For those who missed GrafanaCONline 2021, sessions are available on demand, allowing attendees to engage with past content freely.

Grafana Cloud has introduced a new integration with NSQ, a real-time distributed messaging platform, to enhance full-stack observability by bringing together metrics, logs, and traces. NSQ, which handles billions of messages daily, can now integrate its metrics with Grafana Cloud using a Grafana Agent to convert StatsD metrics into a Prometheus format, facilitating robust deployment and monitoring. This integration provides interconnected dashboards that allow users to analyze NSQ topics and instances, offering both a logical and infrastructure view of metrics. Alerts can also be set up to notify users of critical issues. Grafana Cloud offers a free tier for new users to explore these functionalities, alongside extensive documentation and community support for further assistance.

Integrating load testing into CI pipelines using Grafana k6 and CircleCI can help catch performance issues early and improve application reliability. The process involves writing a k6 performance test script, configuring thresholds for service-level objectives (SLOs), and setting up a CircleCI pipeline with a configuration file that triggers builds upon repository changes. CircleCI orbs, such as grafana/k6, streamline the test execution. Tests can be run locally or on Grafana Cloud for larger or geographically distributed loads, requiring a Grafana Cloud account and API token for cloud tests. The tutorial explains how to store test results as artifacts, schedule nightly builds using crontab syntax, and handle test summaries with k6's handleSummary function. Additionally, it demonstrates the use of k6 extensions, which are Go-based and can be run in a Docker environment, offering further customization for load testing needs. The tutorial emphasizes the importance of defining performance thresholds and using CircleCI's artifact storage for detailed analysis, while also highlighting the ease of starting with Grafana Cloud.

Grafana Labs fosters a no-blame culture where mistakes are seen as opportunities for improvement rather than occasions for assigning fault. Malcolm Holmes, who works on Grafana's release processes, shares his experiences of making errors during software releases, emphasizing how the company's culture supports learning and growth instead of blame. This approach allows individuals to maintain their higher brain functions by staying within the parasympathetic nervous system, which fosters a mindset conducive to problem-solving and future prevention. Holmes highlights the psychological benefits of this culture, noting that it encourages mindfulness and non-judgmental awareness, akin to practices espoused by mindfulness experts like Jon Kabat-Zinn. By focusing on acceptance and constructive responses, both companies and individuals can better adapt to challenges and reduce the likelihood of repeated mistakes.

Dapper Labs, a blockchain gaming studio known for its involvement in the NFT space with projects like CryptoKitties and NBA Top Shot, has adapted to the high demand for digital collectibles by implementing a robust observability stack using Grafana Cloud, PagerDuty, Prometheus, Kubernetes, and Google Cloud Platform. This setup allows the company to effectively monitor over 4 million active series and process about 12 million data points per minute. The adoption of Grafana Cloud has significantly reduced operational challenges related to data retention and enabled the team to engage in proactive problem-solving, catching performance issues before they reach customers. With 240 active users interacting with the observability data, including project and product managers, the open editing culture encourages contributions from all team members, including new hires. Principal SRE Andrew Burian emphasizes the importance of using the observability tools to define strong customer-facing service level objectives, and he plans to further leverage tracing capabilities with Grafana Tempo in the future.

Grafana 8.4 introduces a significant update to the Grafana Loki data source with the default activation of the full-range log volume histogram, enhancing the visualization of log line distribution over a selected time range. Previously, histograms were limited to the first 1,000 returned lines, but now users can explore a comprehensive view that adjusts time intervals automatically, though it can be disabled due to increased query load. This update facilitates detailed log analysis by allowing users to zoom without triggering new queries and to utilize parsers for extracting log levels, which can be displayed as stacked bar charts. Additionally, handling pipeline processing errors is addressed by providing strategies to skip erroneous log lines, ensuring the histogram still renders successfully. The developers encourage feedback from users to further refine this feature, emphasizing its utility and potential improvements.

In the blog post, Fabian Stäber explains how summary metrics in Prometheus are utilized to monitor distributions, such as latencies, using the CKMS algorithm—an efficient method for computing biased quantiles over data streams. The post details how this algorithm works by storing a compressed list of samples to estimate quantiles without needing to store all observations, thus conserving memory. The CKMS algorithm allows users to specify an error margin for quantile approximation, ensuring accuracy by maintaining a strict error bound near the target quantile while relaxing it elsewhere. Although the CKMS algorithm excels in efficiency and memory use, it has limitations, such as its inability to aggregate data across multiple instances and potential inaccuracies in long-tail distributions, which can be mitigated by using histograms. Stäber also highlights that while understanding the CKMS algorithm is not necessary to use Prometheus summaries, it offers valuable insights for avoiding common pitfalls in monitoring.