February 2021 Summaries

Logz.io offers a comprehensive log management solution that utilizes the ELK Stack, enabling users to leverage various log shipping and collector options within this ecosystem. Popular tools like Filebeat, Fluentd, and Fluent Bit are highlighted for their roles in log shipping, particularly in environments like Kubernetes, where Fluentd and its lighter counterpart Fluent Bit are commonly used. Filebeat is recommended for users familiar with their log locations, and its Autodiscover feature can efficiently track dynamic microservices environments. In addition to these open-source options, Logz.io provides integrations with cloud services such as AWS S3, CloudWatch, Azure EventHub, and GCP Stackdriver, allowing seamless log forwarding without the need for installing agents. These integrations are suitable for users already utilizing these cloud services, though they come with additional costs. Logz.io also emphasizes the importance of structured application logging, providing specific logging configurations for languages such as Java, Python, Node.js, and Go, ensuring efficient log shipping to their platform. For users seeking more personalized assistance, Logz.io offers a responsive 24/7 customer support team to help with log shipping and parsing.

In 2020, the enterprise development market shifted significantly towards cloud-native development, embracing microservices architectures with Kubernetes as the dominant platform, largely supported by open source software. The DevOps Pulse survey revealed that a significant portion of infrastructure is now cloud-based, with open source tools like the ELK Stack, Prometheus, Grafana, and Jaeger becoming standard for log management, monitoring, and tracing in Kubernetes projects. Despite their widespread adoption, challenges remain in achieving effective observability, particularly with Kubernetes architecture and scaling open source tools. The OpenTelemetry project aims to address these issues by standardizing data collection methods for telemetry data, enhancing portability and queryability. Companies like Logz.io are capitalizing on these trends by offering scalable SaaS solutions that integrate open source telemetry tools, helping organizations like ZipRecruiter improve performance and scalability while leveraging existing skill investments. The future of cloud-native observability is closely tied to open source tools, which continue to evolve through global community collaboration and vendor innovation, promising improved software reliability and resiliency.

OpenTelemetry 1.0.1 marks a significant advancement towards establishing a standard for observability by focusing on tracing, with future integrations planned for metrics and logs. While the project is still in its early stages, metrics are currently in beta and logging remains in alpha, making them not yet fully usable. This release is pivotal as it impacts tracing with open-source tools like Jaeger and facilitates the transition from OpenTracing. The project emphasizes long-term support, dependency isolation, and backwards compatibility, offering support guarantees for APIs and plugin interfaces. OpenTelemetry's initial release supports Python, Java, .NET, and Erlang, covering traces, spans, events, and attributes. The transition involves integrating OpenTelemetry into systems like Logz.io and re-architecting Jaeger around it, pointing to a collaborative effort within the open-source community to enhance observability tools.

PromQL, the Prometheus Querying Language, is a powerful domain-specific language designed for querying metrics within Prometheus, offering capabilities such as displaying data as graphs or exporting it through an HTTP API. It supports various data types like scalars, range vectors, and instant vectors, and allows for complex nested expressions, enabling sophisticated querying through features like subqueries and various arithmetic and aggregation operators. PromQL's syntax facilitates the grouping and sorting of metrics, making it a concise and efficient tool compared to other domain-specific languages. PromQL's features are enhanced by Prometheus' metric types like counters, gauges, histograms, and summaries, which provide a comprehensive way to monitor and analyze metrics. For those seeking an integrated observability solution, Logz.io offers a Prometheus-as-a-service, combining metrics, logs, and trace analysis using open-source technologies, providing a scalable and low-maintenance approach to cloud infrastructure monitoring.

The text discusses the implications of the Solarwinds hack, known as SUNBURST, emphasizing the need for companies to scrutinize their software products and exposure to potential attack vectors. It highlights the benefits of using Software as a Service (SaaS) over on-premises software, citing the reduced security risks associated with SaaS due to its minimal installation requirements behind security perimeters. The concept of "zero trust" networking, pioneered by Netflix, is recommended as a security measure, alongside the preference for unidirectional data communication with SaaS solutions. Open source software is presented as a more secure alternative due to its transparency and continuous public scrutiny, with frameworks like Apache and CNCF noted for their robust security practices. The text also underscores the importance of penetration testing and bug bounty programs, such as those offered by HackerOne, to enhance security. The author encourages organizations to demand transparency and security assurance from vendors and offers to discuss these topics further on Twitter.

The blog post by Levent Ogut delves into the importance of monitoring web servers for optimal customer engagement in web applications, using Apache as a case study. It highlights the use of Logz.io for monitoring, which involves the integration of the ELK Stack for logging, Prometheus for metrics and alerts, and Jaeger for distributed tracing. The post provides a detailed walkthrough for setting up Apache on Ubuntu 20.04 LTS, configuring Filebeat to ship logs to Logz.io, and using Prometheus with Apache and Node Exporters to collect metrics. It emphasizes the visualization of metrics through Logz.io's Metrics UI and ELK Apps, which offer pre-built dashboards for Apache metrics. The post also discusses Logz.io's "Exceptions" feature, which uses machine learning to analyze logs for anomalies, aiding in troubleshooting and system observability. It concludes by suggesting the potential of application-level monitoring and tracing for deeper insights, inviting readers to explore these capabilities further.

In February 2021, Elastic shifted from an open-source license for Elasticsearch and Kibana to a "source available" license, specifically the Elastic License and the Server Side Public License (SSPL), aiming to restrict competitors like AWS from profiting off their code. This change poses challenges for Managed Security Service Providers (MSSPs), who often use Elasticsearch in their security platforms. While the new licenses prevent the use of Elasticsearch as a service, AWS and Logz.io have responded by announcing an open-source fork of the codebase, potentially offering a community-driven alternative. The new licensing has significant implications for SaaS companies and MSSPs, as it requires releasing their intellectual property if they use ELK stacks in their offerings. Consequently, Logz.io has quickly moved to fork Elasticsearch and Kibana to maintain competitiveness and address the needs of their Cloud SIEM customers, who value interoperability and cost-efficiency.