May 2023 Summaries

Spotify's Backstage, originally developed to boost developer productivity and streamline platform engineering within the company, has gained significant traction since being open-sourced and donated to the Cloud Native Computing Foundation (CNCF). The platform provides developers with tools to create, manage, and explore services within an organization, reducing cognitive load and improving the developer experience through features like software templates and a central service catalog. Backstage's open-source journey has attracted over a thousand adopters across various industries, including banking and healthcare, and has become a part of the emerging trend of Internal Developer Portals (IDP), which facilitate collaboration and communication between developers. The open-source model has allowed for rapid community-driven enhancements, such as a dark mode feature that was implemented within 24 hours, and continues to drive innovation as new use cases and verticals emerge. Spotify aims to support engineers and reduce cognitive load by growing the Backstage community and enabling productivity engineering across industries.

Observability challenges in modern organizations, such as engineering expertise shortages and data overload, are being addressed through AI innovations like the new Alert Recommendations feature in the Logz.io Open 360™ platform. This feature leverages supervised machine learning to model the actions of experienced users and recommend optimal troubleshooting paths for software issues, thereby reducing mean time to resolution (MTTR) and enhancing team efficiency. The system tracks user actions during investigations, applies clustering algorithms to identify effective patterns, and ranks these actions through a learning algorithm to offer tailored recommendations. As more organizations utilize this alerting system, the recommendations become increasingly refined, exemplifying the potential of AIOps to automate and streamline observability tasks, enabling teams to work smarter and faster in dynamic cloud environments.

Endeavor Streaming's platform development team, led by Vice President Claudio Ferrete, is focused on balancing the operation and development of its digital video platform by maintaining comprehensive visibility into its cloud infrastructure. To support this mission, the team employs Logz.io for various observability practices, including metrics management, which allows them to monitor and analyze their platform's performance at both macro and micro levels. This approach is essential due to the platform's complexity and the diverse capabilities of its global customer base. Logz.io's tools, including pre-built dashboards and extensive documentation, facilitate the integration of telemetry into Endeavor's codebase, enabling a swift transition from experimentation to production. Additionally, Endeavor plans to migrate its current distributed tracing system to Logz.io's solution to further enhance its observability practices.

Kubernetes adoption is rapidly increasing, surpassing even cloud adoption rates, as companies seek the agility and flexibility it offers for deploying critical applications. However, this comes with increased complexity in observability and security, as the abstraction layers in Kubernetes make it difficult to track what is running in each cluster, node, or pod. The 2023 DevOps Pulse Report highlights that nearly half of organizations struggle with Kubernetes observability and security, with many planning to implement unified monitoring models. Proper knowledge of Kubernetes deployment, coupled with robust observability setups, is crucial to address these challenges. Cost management is also a significant factor, as companies aim to balance expenses with availability and security. Logz.io's Kubernetes 360 solution, enhanced by integration with Aqua Security's Aqua Trivy for vulnerability scanning, aims to help enterprises improve their Kubernetes observability and security practices.

The 2023 DevOps Pulse survey highlights several significant trends in the observability landscape, including an increase in mean time to recovery (MTTR), challenges with Kubernetes observability and security, and the convergence of observability and security practices. The survey reveals a concerning rise in MTTR, with over 75% of organizations experiencing recovery times exceeding several hours, a trend attributed to the complexity of cloud technologies and siloed telemetry data. Kubernetes adoption presents major observability and security challenges, with nearly 50% of respondents identifying it as their main obstacle. To address this, organizations are advised to deepen their understanding of Kubernetes deployment. Moreover, the survey indicates a growing integration of observability and security, with 46% of observability teams being accountable for security and a high percentage planning to implement a unified model. As organizations transition to cloud-native technologies, new security challenges have emerged, underscoring the necessity for integrated security solutions.

Logz.io has introduced a new Home Dashboard within its Open 360™ platform, designed to provide users with a comprehensive overview of their observability environment, including logs, metrics, and traces, as soon as they access the platform. This feature aims to streamline the user experience by enabling quick access to critical telemetry data and insights, categorized by timeframe, event type, count, and severity level. Users can seamlessly investigate events through associated logs, facilitating efficient data exploration and troubleshooting. The Home Dashboard also offers easy navigation to logs and metrics dashboards, allowing users to quickly access recently used or favorite dashboards, thereby enhancing the overall process of managing observability data in complex cloud infrastructures.

Logz.io undertook a significant data volume optimization initiative, successfully reducing their internal log volume by 50% from November 2022 to February 2023, achieving an average daily volume of 1.5 TB. This effort was driven by the need for cost efficiency in observability practices, focusing initially on identifying and eliminating low-value logs using their Data Optimization Hub. The process involved setting drop filters for infrequently used logs, collaborating with development teams to manage obsolete logs, and refining the generation of heavy logs by removing unnecessary information. The company emphasized the importance of ongoing monitoring and alerts to maintain reduced log volumes and costs, highlighting their AI-powered observability tools and inviting users to explore their Data Optimization Hub through a free trial.