May 2020 Summaries

Security Information and Event Management (SIEM) tools play a crucial role in modern cybersecurity by collecting data from various systems to identify patterns and generate actionable intelligence. While traditional SIEM solutions faced challenges such as expensive on-premise deployment, limited integrations, and a focus on external threats, modern cloud-based SIEMs address these issues through scalable data handling, compliance with regulations, and enhanced threat detection by considering both internal and external threats. These advancements include leveraging User and Entity Behavior Analytics (UEBA) for detecting anomalies and integrating with enterprise programs to provide meaningful alerts and recommendations for action. Modern SIEMs are more accessible to non-security experts, offering relevant alerts and automated responses, thus enabling businesses to predict security threats, conduct in-depth data analysis, and enhance incident response capabilities.

In the context of managing cloud environments, the decision between using open source or proprietary observability solutions is often influenced by the total cost of ownership (TCO), which extends beyond the initial price tags. Open source solutions might appear cost-effective with no upfront fees, while proprietary solutions are usually priced based on the scale of data collected. However, the true cost encompasses factors like infrastructure expenses and the labor required to manage and maintain the system. For instance, a hypothetical cloud environment generating 100 GB of log data daily, needing a seven-day retention for incident analysis, involves costs related to Amazon infrastructure and the time commitment of DevOps engineers, where each hour of their work is valued at $100. Additionally, aspects such as security features, compliance requirements, and the impact on staff productivity, where time spent on non-revenue-generating activities reduces business value, further complicate the cost analysis. Despite the complexities, understanding these cost components can lead to a more financially prudent design of an observability stack, taking into account both immediate and long-term financial implications.

As organizations transition to cloud-based development, monitoring their computing environments becomes more complex, prompting consideration of building custom monitoring solutions using open source tools. Open source monitoring provides flexibility, scalability, and cost-effectiveness, as it avoids licensing fees and enables shared security and support costs. Building an in-house solution involves careful planning and resource allocation across the web, application, and database tiers, with considerations for storage, buffering, and data retention. While tools like Grafana can ease the development of custom dashboards, adopting a DevOps methodology and integrating event-driven architecture can improve scalability and efficiency. However, the notion of "free" in open source isn't synonymous with "zero cost," as it entails ongoing maintenance and potential additional expenses. Organizations must weigh the benefits against the challenges of technical debt and resource allocation when deciding on a custom solution versus managed services.

Elasticsearch and MongoDB are two leading distributed datastores designed for handling NoSQL data, each with distinct strengths and ideal use cases. Elasticsearch, an open-source search engine built on Apache Lucene, excels in full-text search, log analytics, and anomaly detection, offering features like distributed search, RESTful interfaces, and integration with tools like Kibana. It is particularly suited for scenarios where complex search queries and geo-search capabilities are crucial. Conversely, MongoDB is a document-oriented database known for its scalability and high throughput in handling large datasets, making it optimal for applications requiring CRUD operations without the need for rapid full-text search. Written in C++, MongoDB provides features such as built-in security, schema-less design, and support for various data models, making it prominent in document store databases. Both technologies offer robust backup and recovery options, but they differ in handling relational data and data storage architecture, with Elasticsearch focusing on immutable Lucene segments and MongoDB utilizing a memory map file for efficient data management. While Elasticsearch is written in Java and offers a multitude of client support languages, MongoDB also provides extensive language support and operates under a Server Side Public License (SSPL), with advanced features available in its enterprise edition.

Modern cloud environments generate vast amounts of log data, making it costly to index all of it, thus necessitating effective cost management strategies without compromising observability. Logz.io offers solutions like Archive & Restore, Log Patterns, and Drop Filters to manage these costs efficiently. Archive & Restore allows businesses to store logs in inexpensive storage like an S3 bucket, enabling access and reingestion if needed, ensuring that crucial data is not permanently lost. Log Patterns help identify noisy logs by clustering similar logs and organizing them by frequency, which aids in recognizing which logs are using significant resources without frequent use. Drop Filters further optimize cost management by filtering out rarely needed logs before indexing, reducing unnecessary data processing and storage costs. Together, these features enable businesses to maintain a balance between cost efficiency and comprehensive data analysis, providing the flexibility to reingest data if it becomes crucial for troubleshooting future incidents.

FinOps extends beyond cloud cost management to include observability and infrastructure monitoring, where out-of-the-box solutions often fall short in features and affordability, prompting organizations to develop their own cost-effective monitoring systems using a mix of in-house talent and open-source libraries. A comprehensive monitoring application typically comprises a web interface, an application tier for data processing, and a database tier for data storage, with the flexibility to incorporate cloud and on-premises technologies like event queues. Cost management in infrastructure monitoring involves ensuring solutions are appropriately scaled to prevent overspending, utilizing cloud-native architecture analysis tools like AWS Well-Architected Framework and Azure Advisor to optimize resource provisioning. Multicloud cost management adds complexity as it requires careful monitoring and scaling of resources across different platforms, emphasizing the need to spin down idle or over-provisioned servers and regularly reassess non-production environments. Hidden costs, such as data retrieval and maintenance of open-source solutions, must be managed to ensure the sustainability of custom infrastructure monitoring solutions, which can effectively leverage modern FinOps best practices for native, hybrid, and multicloud environments.

Earlier this month, the Observability All-Star program was announced to honor DevOps industry professionals and teams, with Justin Smith, a DevOps Engineer at MIM Software, being recognized as the first winner for 2020. MIM Software offers imaging solutions across several medical fields, and Justin's diverse IT career, spanning roles from technical writer to DevOps engineer, exemplifies his commitment to breaking down silos and enhancing collaboration. In his current role, he leads DevOps initiatives, focusing on scaling engineering workflows and organizing infrastructure workgroups while spearheading MIM’s observability program. He has successfully implemented Logz.io to address challenges in managing an ELK stack for log management, thereby enhancing the team's responsiveness to production issues. Logz.io has been praised for its ease of setup, features like machine learning, and excellent customer service, offering a cost-effective alternative to platforms like Splunk for server log analytics and monitoring.

The evolution of monitoring systems from basic infrastructure monitoring to advanced distributed tracing and observability solutions is driven by the need to understand complex software environments and address business challenges in digital-first organizations. Initially, software agents collected metrics and logs to diagnose issues, but as applications scaled, log analytics emerged as a centralized debugging tool. This gave rise to Application Performance Monitoring (APM) solutions, which evolved to include business metrics. Open source initiatives, led by major tech companies like Google, Twitter, and others, facilitated the development of standardized protocols and tools for distributed tracing, such as Zipkin, Jaeger, and Skywalking. OpenTracing and OpenCensus merged to form OpenTelemetry, aiming to provide comprehensive instrumentation standards. Despite the rapid expansion in open source options for observability, the field remains in flux, with ongoing efforts to improve maturity and integration across various systems.

Azure Kubernetes Service (AKS) is a managed Kubernetes platform introduced by Microsoft in October 2017 that simplifies the deployment of Kubernetes clusters without the need to maintain the underlying infrastructure. This tutorial serves as a guide for setting up a proof of concept for AKS, detailing the step-by-step process of using the Azure CLI to configure an Azure subscription, create a resource group, and deploy an AKS cluster with a Kubernetes API version. The tutorial also covers cloning the Kubernetes configuration (kubeconfig) to a local machine, accessing AKS from a local terminal, and deploying a demo Nginx application on the cluster. By utilizing AKS, organizations can efficiently manage and scale containerized applications, leveraging Kubernetes' orchestration capabilities while minimizing infrastructure management overhead.

Securing Kubernetes environments requires continuous monitoring and threat detection due to inherent vulnerabilities that cannot be entirely eliminated, even with best practices. Effective runtime security involves using tools like Falco, an open-source threat detection engine that leverages Linux kernel instrumentation to detect suspicious behavior by monitoring system calls. Falco's integration with Logz.io Cloud SIEM enhances threat monitoring by allowing users to collect, index, visualize, and analyze incidents using the ELK stack. Cloud SIEM provides enriched data and dashboards to filter false positives and prioritize real threats, with alerts and notifications sent to platforms like Slack or email. This approach is exemplified by detecting sophisticated attacks, such as reverse shells executed within containers, where Falco can identify netcat activity and Cloud SIEM applies high-order rules to recognize command shell initiation attempts. Overall, a combination of audit logs, operational logs, and advanced threat detection tools helps maintain the security of Kubernetes environments.