July 2021 Summaries

The text discusses the evolution of Security Information and Event Management (SIEM) systems in the context of cloud computing, emphasizing the transition to cloud-native architectures as necessary to meet modern security needs. As traditional SIEM vendors expand into areas like Security Orchestration, Automation, and Response (SOAR) and Endpoint Detection & Response (EDR), the text questions whether these expansions benefit users, suggesting that they may limit interoperability and choice. The Cloud Native Computing Foundation (CNCF) defines cloud-native technologies as empowering organizations to build scalable applications in dynamic environments using containers, microservices, and APIs, which enhances resilience and manageability. Logz.io exemplifies this approach by operating on a microservices architecture within public clouds, leveraging the latest technologies from providers like AWS and Azure, and contributing to open-source projects such as OpenSearch. The company emphasizes rapid release cycles and resilience through geographically distributed infrastructure, aligning with the CNCF's principles to support a robust and flexible cloud-native SIEM solution.

Logz.io has formed a partnership with Microsoft to integrate its platform directly within the Azure Console, allowing Azure users to quickly and effortlessly begin monitoring their workloads with open-source technologies. This native integration, accessible through the Azure Marketplace, enables users to ship log data to Logz.io without needing to deploy new code, offering features like setting up sub-accounts, SSO with Active Directory, and selecting specific logs for analysis. The collaboration accelerates application response times, increases service availability, and centralizes security event monitoring across Azure environments. It also offers a streamlined, cost-effective "pay for what you use" pricing model, simplifying the initiation and management of observability tools for Azure developers. This integration represents a significant advancement in cloud monitoring, facilitating seamless access to Logz.io's insights and enhancing the ability of engineering and DevOps teams to optimize their applications.

Logz.io has introduced a new open-source RemoteWrite SDK to facilitate direct metric shipping from applications to Logz.io without routing through Prometheus first, supporting languages like Golang, Python, and Java, with plans to expand to .NET and Node.js. This initiative aims to enhance the ease of sending metrics from code for the broader Prometheus and DevOps communities, leveraging integrations with frameworks such as Thanos, Cortex, and M3DB. The Python SDK, for instance, is built on OpenTelemetry, requiring installations like the Snappy C library and OpenTelemetry exporter, and supports various metric instruments including counters and valuerecorders. The integration underscores Logz.io's commitment to OpenTelemetry and aims to support developers in creating and managing metrics more efficiently, with further SDKs and language support anticipated.

Logz.io has introduced a new RemoteWrite SDK designed to facilitate the direct transmission of custom metrics from applications written in Golang, Python, and Java to Logz.io using the RemoteWrite protocol, bypassing the need for initial routing through Prometheus. This open-source tool aims to enhance the ease with which developers can send metrics directly from their code, benefiting both the Prometheus and broader DevOps communities. The SDKs are compatible with frameworks like Thanos, Cortex, and M3DB, with the Java version utilizing a Micrometer registry and the Go and Python versions based on OpenTelemetry. Logz.io plans to expand language support by adding .NET and Node.js, accompanied by tutorials to guide users through the process, with a focus on making the integration straightforward. The company encourages users to stay informed about updates on custom metrics and related topics by subscribing to their blog.

Monitoring AWS Lambda functions effectively involves understanding and tracking key metrics such as invocation, performance, and concurrency metrics to identify and address potential issues. The process includes using Prometheus to collect metrics from CloudWatch and forwarding them to Logz.io for scalable storage and analysis, allowing for unified monitoring and troubleshooting with log analytics. By setting up dashboards and alerts in Logz.io, teams can visualize and manage their Lambda metrics more efficiently, with alerts notifying users of issues like increased error counts. This setup streamlines the process of correlating metrics with logs, enhancing the ability to debug and resolve problems swiftly. The choice between using Prometheus or Logz.io for monitoring depends on a team's resources and preferences, with Logz.io offering a managed service that unifies metrics, logs, and traces in one platform.

eBPF (extended Berkeley Packet Filter) is emerging as a transformative technology for Linux kernel-level instrumentation, offering the potential for zero-code observability and auto-instrumentation, particularly in Kubernetes and microservices environments. Unlike traditional manual instrumentation, eBPF allows for the collection of telemetry data without modifying application code by running code within the kernel through hooks and probes. This feature enables it to capture a wide range of data, including network packets, system metrics, and encrypted traffic, providing a unified observability framework. Supported by major tech companies and integrated into projects like OpenTelemetry and Pixie OSS, eBPF is being adopted in large-scale operations, such as Netflix's network observability, and is poised to revolutionize the industry by simplifying observability and enhancing performance with minimal resource overhead. As it gains popularity and extends to platforms like Windows, eBPF is expected to drive significant advancements in observability, security, and networking, echoing the impact of Docker on application containerization.

OpenSearch has reached general availability with version 1.0, marking a significant milestone for this new open-source initiative, which emerged after Elastic's decision to change the licensing for Elasticsearch and Kibana. This release offers a pure Apache 2.0 open-source equivalent of the popular Elasticsearch and Kibana projects, enhanced with plugins from Open Distro for Elasticsearch, and introduces new features like data stream support for OpenSearch Dashboards, ARM64 support, and plugins for distributed tracing analytics. AWS played a crucial role in forking the project, and the release is designed for production deployments, available both as a standalone tool and for embedding into other products. Looking ahead, the roadmap includes several minor releases and a major version 2.0 release in early 2022, promising further advancements in cross-cluster replication, anomaly detection, and more.

CrowdStrike has established itself as an innovator in endpoint protection over the past decade, particularly through its deep data collection capabilities that have helped uncover numerous security mysteries. Collaborating with Logz.io, an observability platform, the partnership aims to enhance threat detection by integrating CrowdStrike's endpoint data into Logz.io's Cloud SIEM. This integration facilitates the analysis and correlation of threat data, offering customizable alerts and dashboards to fit user-specific needs. The alerts notify users of malicious files, policy changes, and other suspicious activities, enabling SecOps teams to contextualize security signals and take corrective actions. CrowdStrike's Falcon SIEM Connector plays a key role by collecting data and streaming it through Fluentd to Logz.io, where operational dashboards provide insights across multiple instances. The collaboration underscores the mutual commitment to providing added value to shared customers through more effective threat detection and response capabilities.

OpenSearch emerged as a community-driven response to the relicensing of Elasticsearch, aiming to serve as a compatible yet independent alternative while incorporating all former Open Distro plugins and introducing new features like Docker images and visualization capabilities. Released as version RC1 (1.0.0) in June 2021, OpenSearch is feature-complete but not production-ready, and it requires Docker for installation. The setup process involves pulling Docker images for OpenSearch and OpenSearch Dashboards, ensuring there are no port conflicts with Elasticsearch or Kibana, and deploying nodes using a docker-compose.yml template. OpenSearch's architecture allows for customization through plugins, continuing the legacy of Open Distro by integrating both existing and new plugins for enhanced functionality. Future tutorials promise to delve deeper into advanced topics such as clusters, mapping, and queries.