April 2022 Summaries

Elasticsearch has been a leading solution for log management and analytics, but its limitations in handling scalable logging volumes and cloud-native operations have prompted Slack to develop KalDB, a new project discussed by Suman Karumuri. KalDB, which shares its foundation with Apache Lucene, is designed to automatically scale with logging volume, handle field conflicts on read, and offer multi-tenancy with isolation, which are improvements over Elasticsearch. The system is cloud-native, easing operations on platforms like Kubernetes, and aims to unify events, logs, and traces (ELT) data under a single system for simplified infrastructure and more expressive queries. The project also enhances tracing capabilities for software and business processes, and while it shares roots with the OpenSearch project, KalDB is intended to become open-source, with its initial source already available on GitHub.

Logz.io's 2022 DevOps Pulse Report highlights the rapid evolution and growing maturity of cloud and DevOps practices over the past five years, emphasizing trends such as increased adoption of observability strategies, with 77% of organizations actively engaged in this area. However, challenges persist, including a rise in mean time to recovery (MTTR) during production incidents and difficulties in monitoring Kubernetes and microservices, cited by 52% of respondents. The report notes the nascent state of distributed tracing, though its importance is increasingly recognized, with 80% of respondents rating it crucial to their observability strategy. Additionally, the survey reveals concerns over the growing cost and complexity of maintaining effective observability, with tool sprawl and data volumes being significant obstacles. Despite these challenges, the report underscores the ongoing development of DevOps skills and tools, with a focus on improving visibility and managing observability costs.

Slack's need for a more efficient metrics storage engine than Prometheus led to the development of MACH, which promises significant performance improvements, including 10 times the write throughput and 3 times the read throughput. Created by a team including Slack and several academic and industry partners, MACH addresses the challenges of handling metric data's unique time and space dimensions through features like tiered storage, a staged ingestion pipeline, and multi-value time series ingestion. Unlike Prometheus, which uses a fixed compression mechanism, MACH allows for selecting compression algorithms tailored to specific data types, enhancing efficiency and preventing data explosion. The architecture of MACH, which supports fast data recovery and efficient parallel ingestion, is designed to scale with the high core counts of modern CPUs, offering substantial boosts in operational efficiency. While the code for MACH is not yet publicly available, a proof of concept is promised for future open-source release, and further details are available in the research paper published in CIDR.

On March 29, 2022, VMware disclosed a critical vulnerability within the Spring Java framework, known as "Spring4Shell" or "SpringShell," officially tracked as CVE-2022-22965, which allows for full remote code execution (RCE) through class injection. This zero-day vulnerability, found in Spring Core's "data binding" mechanism, has raised significant concerns over potential malicious attacks on applications using the library, as it can be exploited via specially crafted HTTP requests. Comparisons have been drawn to the infamous Log4Shell vulnerability due to Spring Core's widespread use. Logz.io provides its Cloud SIEM customers with out-of-the-box detection rules to identify possible Spring4Shell exploitation by scanning for webshell activity in access logs, allowing for real-time monitoring and protection against attackers. The vulnerability's ease of exploitation in production environments necessitates heightened awareness and security measures, such as those offered by Logz.io's Cloud-based SIEM with a free 14-day trial available for new users.

Centralized observability teams are crucial in modern organizations to manage the complexity and scale of telemetry data effectively, as highlighted by Suman Karumuri from Slack. These teams, which can go by various names like Shared Services or Platform Engineering, are responsible for maintaining infrastructure, ensuring standardization in data formats, and disseminating knowledge across engineering teams. Without centralized guidance, different teams might adopt divergent logging formats, complicating cross-team event correlation and potentially leading to duplicated efforts and multiple monitoring stacks. Observability is viewed as a data analytics problem, where understanding system behavior involves gathering and ingesting varied data types to answer critical questions. Furthermore, observability encompasses legal compliance issues, such as GDPR, by ensuring sensitive data is not exposed in logs. The right technical stack, whether ELK Stack, OpenSearch, or others, should be chosen based on an organization's specific needs, making observability not just a backend concern but a comprehensive approach covering business processes and user interactions.