July 2020 Summaries

At OpenObservability 2020, Andre Boutet, Senior Director of Cloud Operations and Services for OneSpan, discussed the significance of observability beyond mere uptime and availability, highlighting its role in supporting the entire development lifecycle. Under Boutet's leadership, OneSpan's cloud operations team transitioned on-premise products to cloud-native environments, emphasizing shared ownership and self-service capabilities to enhance developer experience and operational efficiency. The organization leverages open-source technologies for their flexibility and plug-and-play nature, allowing for seamless integration and customization, which is crucial for observability and monitoring efforts. By using tools like Logz.io, OneSpan aims to improve internal responses, streamline code delivery, and provide C-level executives with insights into business operations, ultimately enhancing the end-user experience. Boutet emphasizes the importance of not being tied to specific vendors and praises the ability of open-source platforms to future-proof their technology infrastructure.

Logz.io's integration with OpenVAS, a widely-used open-source vulnerability scanner, enhances security operations by utilizing logs from OpenVAS to identify and prioritize vulnerabilities through Logz.io Cloud SIEM, a security information and event management system based on the ELK Stack. This integration allows security teams to use tools like Filebeat to ship OpenVAS logs to Logz.io, where they are parsed, visualized, and analyzed in Kibana to provide insights into security vulnerabilities. Logz.io offers pre-built dashboards and rules that help focus on critical security events, providing a comprehensive overview and enabling drill-down investigations into specific vulnerabilities. These capabilities are enhanced by automatic alerts sent through various endpoints when significant security events are detected, and the integration supports a broader ecosystem of security tools for a unified security management approach.

Solr and Elasticsearch are two prominent open-source search engines built on Apache Lucene, each with unique strengths and applications. Solr, a mature platform, excels in enterprise-directed text searches and integrates well with big data tools like Hadoop, making it ideal for handling static data in enterprise settings. In contrast, Elasticsearch, though newer, has rapidly gained popularity due to its scalability, JSON-based structure, and suitability for time-series data and modern web applications, particularly in log management. While both engines offer robust search and indexing capabilities, they differ in deployment and scalability features, with Elasticsearch favoring horizontal scaling and Solr involving more complex cluster management. Although Solr has a strong community foundation, Elasticsearch's development is more company-driven, resulting in comprehensive documentation and resources that surpass Solr's currently outdated materials. Both engines are easy to start with, but Elasticsearch's design is more conducive to production environments and large-scale applications.

OpenObservability Talks, a continuation of the OpenObservability Conference, was launched on June 25th, aiming to explore open-source observability through talks, panels, and discussions. The first episode featured Jack Neely from 42 Lines, who shared his insights on application monitoring with Prometheus, offering tips such as creating dynamic threshold alerts and effective SLO alerts. These sessions are available for live viewing on the last Thursday of each month via Twitch and YouTube, with recordings accessible on YouTube and various podcast platforms afterward. The next episode is scheduled for July 30th, and the organizers welcome ideas for future content submissions via a Google form or email.

Gedalyah Reback's article explores the distinct roles and capabilities of Prometheus and Nagios in infrastructure monitoring, highlighting the importance of metrics like mean-time-to-detect (MTTD) and mean-time-to-repair (MTTR) for SaaS companies. Prometheus is praised for its ease of integration and use in monitoring application functionality, with features like a built-in database, PromQL query language, and compatibility with visualization tools like Grafana. In contrast, Nagios excels in monitoring application networks and security, offering multiple products like Nagios XI, Log Server, Network Analyzer, and Fusion, each tailored to specific monitoring needs. Prometheus is noted for its extensive open-source community support and flexibility, while Nagios is commended for its scalability and customization capabilities. Despite their differences, the article suggests that using both tools together could provide a more comprehensive monitoring solution for DevOps teams, as demonstrated by Runtastic's migration from Nagios to Prometheus due to configuration challenges and false positive alerts.

Monitoring investments today are increasingly aligned with automation, a key objective of DevOps to reduce manual work and enhance scalability in application development and operations. Despite the benefits, many organizations find themselves overwhelmed with a multitude of automation tools due to siloed teams and specific domain needs, as highlighted by a 2018 EMA survey indicating that 56% of enterprises face this challenge. Robotic Process Automation (RPA) and Application Release Orchestration (ARO) are notable areas of interest, with RPA focusing on automating user interface interactions with legacy systems, potentially adding technical debt, and ARO streamlining application deployment and management of CI/CD pipelines. While traditional tools like Jenkins and open-source alternatives like Spinnaker are popular in CI/CD processes, newer solutions such as Apollo, Keptn, and Harness offer more integrated or event-driven automation approaches. Harness exemplifies a commercial option with native integrations for continuous verification of changes, contrasting with the more customizable but complex open-source alternatives. The evolving landscape continues to see initiatives like Dynatrace's Keptn working towards standardized automation, although sophisticated AI and ML integration remains limited with the exception of a few platforms.

As the popularity of distributed systems continues to rise, Kubernetes has emerged as a key tool for orchestrating container-based environments, with Prometheus being a leading open-source monitoring solution for these systems. The choice between manually deploying Prometheus using Docker Hub, static YAML manifests, and kubectl, versus using Helm charts for an automated deployment, hinges on the level of control versus convenience one desires. Manual deployment offers complete control over the configuration, allowing direct influence over every component, though it can be time-consuming and complex. On the other hand, employing Helm, a popular package manager within the Cloud Native Computing Foundation (CNCF) ecosystem, streamlines the process substantially, enabling rapid deployment of comprehensive monitoring stacks with minimal commands, albeit with less granularity in configuration. Developers must weigh the trade-offs between hands-on management and higher-level automation, considering factors such as ease of use, maintainability, and security when choosing their deployment strategy.

Chaos engineering is a proactive approach adopted by companies like Netflix, Amazon, and Google to enhance system resilience by deliberately injecting failures into systems to identify and rectify vulnerabilities. This method is particularly crucial for container orchestration tools like Kubernetes, which are essential for managing complex application ecosystems transitioning from monolithic to microservices architectures. While Kubernetes offers a robust framework for managing containerized applications, chaos engineering allows organizations to stress-test their systems, ensuring they can handle real-world challenges such as network failures, resource exhaustion, and traffic surges. Effective chaos testing helps identify weaknesses in Kubernetes deployments, bolstering security and performance by simulating and analyzing diverse failure scenarios. By integrating tools like Logz.io for monitoring, organizations can visualize and interpret chaos testing results, ensuring that potential issues are addressed before impacting end-users. This strategy not only fortifies the system's stability and security but also emphasizes the importance of redundancy and failure mechanisms to maintain continuous uptime, which is critical for sectors like streaming services and e-commerce.

OpenTelemetry is emerging as a unified standard for observability signals, seeking to combine the strengths of previous initiatives like OpenTracing and OpenCensus to create a comprehensive framework for distributed tracing. The discussion at OpenObservability, featuring representatives from projects like Zipkin, Jaeger, and Skywalking, highlighted the challenges and opportunities in standardizing tracing practices within the industry. OpenTelemetry aims to standardize data formats and instrumentation, offering a vendor-neutral API and a standard SDK, which could simplify the integration of distributed tracing tools. This effort is seen as a significant step forward, although it also presents challenges in balancing existing approaches and accommodating diverse technologies. The ongoing development includes creating agents and collectors that adhere to the new standards, facilitating easier adoption and more sophisticated telemetry practices. Despite the complexities involved, the consensus is that OpenTelemetry has the potential to reshape the observability landscape by providing a standardized approach that supports both legacy and modern systems.