September 2019 Summaries

Kubernetes has become a crucial part of modern IT infrastructure, but its implementation and maintenance require careful planning and expertise to realize its potential value. After the deployment phase, DevOps teams must focus on challenges such as continuous maintenance, where regular system updates can both improve and disrupt a cluster's functionality. The Horizontal Pod Autoscaler (HPA) is a powerful tool for managing service load, but if improperly configured, it can lead to unintended business impacts. High availability features allow for risk distribution across different locations, minimizing downtime risks. Security is crucial and should be continuously reviewed and integrated into the cluster's lifecycle, with automated processes like DevSecOps enhancing protection. Monitoring is essential for ensuring cluster health, but traditional alert systems need adjustments to avoid unnecessary human intervention, focusing instead on verified issues. Successfully managing Kubernetes requires understanding its intricacies to avoid negative returns on investment, making it vital for organizations to have a thorough continuation plan and a robust continuous deployment process.

Kubernetes, a leading orchestration tool for managing containerized applications, offers various implementation options and supports numerous configuration settings, facilitating automated environment maintenance and streamlined deployment processes. However, as Kubernetes evolves, it faces increasing security threats and vulnerabilities, necessitating robust security measures. A recent audit by the Cloud Native Computing Foundation highlighted several vulnerabilities that attackers could exploit, stressing the importance of securing Kubernetes clusters. Effective security strategies involve protecting Kubernetes pods, nodes, and networks through minimalistic operating systems, communication route configurations, namespaces, and real-time monitoring tools. To mitigate risks, Kubernetes clusters require comprehensive auditing and logging of all activities, as well as strict access control through role-based access control (RBAC) and data encryption. DevOps teams must remain vigilant and proactive in addressing vulnerabilities, keeping abreast of the latest security practices and ensuring that telemetry data is continually monitored to detect and respond to potential threats.

Monitoring systems collect performance data across various targets, and incident management systems utilize this data to detect, prioritize, and resolve disruptions in service operations. This blog post discusses the integration of these systems to enhance proactive, real-time incident responses, thereby improving business outcomes. By combining monitoring alerts with other inputs, incident management creates a comprehensive view that aids in quick diagnosis and resolution. The integration reduces mean time to resolution (MTTR) through automation, improves monitoring algorithms, and facilitates collaboration among development, QA, operations, and ITSM teams. Additionally, it minimizes alert fatigue, reduces support costs, and supports scalability. Popular incident management systems like Jira Service Desk, Mantis BT, PagerDuty, VictorOps, Fresh Service, and OpsGenie offer various features such as real-time updates, collaboration tools, and customizable alerts. A case study of Logz.io and PagerDuty integration illustrates how systems can work together to efficiently manage alerts and notifications. The synergy achieved by integrating monitoring with incident management systems is emphasized as crucial for maintaining performance SLAs and minimizing downtime in today's competitive market.

Kenshoo's engineering team developed a proactive log monitoring and alerting system using Logz.io to maintain the functionality and credibility of their marketing platform, which integrates with over 100 third-party systems. By implementing this system, they aimed to catch and address issues before they impact customers, thereby preventing downtime that could result in substantial financial losses. To enhance efficiency, they created an IntelliJ/Jetbrains Plugin that allows developers to set up alerts directly from their code, fostering a culture of full cycle ownership and testing within the development team. This plugin was designed to simplify the alert creation process and was made community-driven, with plans to open source it, allowing other developers to benefit from streamlined log-based alerting. The team believes this approach not only improves their internal processes but can also aid others in achieving similar operational goals.

Log management has evolved significantly due to modern architectures, including cloud computing and distributed environments, which have introduced challenges like extensive logging pipelines that are costly and time-consuming to manage. Despite the general disdain for logs, their importance in providing system visibility is widely acknowledged, leading to the development of methodologies to enhance logging efficiency and usability. Structured logging, which standardizes logs across applications, has emerged as a best practice to improve observability, with frameworks like the Elastic Common Schema playing a central role. The concept of "shifting left" in logging encourages developers to integrate logging early in the development process to ensure comprehensive visibility throughout the application lifecycle. To address the financial and practical costs of logging, some organizations adopt a "log less" approach, focusing only on critical components to mitigate the volume of data generated. These strategies, combined with automation, aim to optimize logging workflows and help teams handle the vast quantity of log messages produced by modern systems, enhancing the overall effectiveness of monitoring systems.

The public cloud revolution initiated by Amazon Web Services (AWS) in 2002 has transformed IT service deployment and delivery, leading to a significant growth in the Infrastructure as a Service (IaaS) market, which was valued at $41.4 billion in 2019 and projected to reach $81.5 billion by 2022. The market is dominated by AWS, Microsoft Azure, and Google Cloud Platform, with IBM, Oracle, and Alibaba Cloud serving niche areas. While the major providers offer similar IaaS products, the choice of provider often hinges on organizational needs, geographic requirements, and the desire to avoid vendor lock-in through a multicloud strategy, as evidenced by 84% of respondents in the Rightscale 2019 State of the Cloud Report employing such a strategy. The blog post outlines fifteen guidelines for evaluating cloud service providers, emphasizing considerations like compute instance sizing, operating system support, geographic location, pricing, availability SLAs, data storage and security, database services, big data analytics, hybrid storage, and application support, including DevOps tools, modern app frameworks, monitoring, IoT, and AI-based services. It underscores the need for organizations to clarify their specific use cases and requirements when choosing a cloud provider, also noting the importance of considering the provider's network of partners to enhance offerings.

Elasticsearch, a versatile distributed search engine, has evolved into a comprehensive NoSQL storage and analytics tool, requiring careful management to optimize performance and reliability, particularly through effective index and shard management. Proper configuration of indices, including sharding and replication, is crucial for maintaining stability and resource efficiency, with features like the Rollover and Shrink APIs addressing index management for time-series data and the Freeze API allowing resource optimization for rarely accessed indices. The Index Lifecycle Management (ILM) feature, introduced in Elasticsearch 6.7, automates index transitions through different lifecycle phases, enhancing resource allocation and system performance. Additionally, creating explicit mappings and adhering to the Elastic Common Schema can improve data consistency and facilitate easier data search and visualization across diverse data sources. For those who prefer not to directly manage these tasks, solutions like Logz.io offer fully managed OpenSearch clusters, enabling users to leverage the ELK Stack's capabilities without handling index management.

Logz.io has introduced several new features and enhancements aimed at optimizing logging pipelines and improving user experience, with highlights including the Drop Filters feature, which allows users to selectively decide which logs to store and index, thereby managing logging quotas more efficiently. The Reports feature automates the generation of regular reports from Kibana dashboards, while the revamped Archive/Restore functionality offers a streamlined process for data backup and retrieval using S3 storage. Additionally, the platform now supports IAM role authentication for S3, providing a more secure method for accessing AWS buckets, and has expanded its security analytics by integrating with the Zeek network analysis tool. The company is dedicated to continuous improvement, seeking user feedback and promising new features and integrations in the coming months, and is actively engaging with the community, including participation in events like AWS re:Invent.

Grafana is a widely acclaimed open-source visualization tool used by DevOps engineers and SREs for monitoring time-series data through its aesthetically pleasing and customizable dashboards. The process of building these dashboards can be challenging, especially for beginners, but the clarity and efficiency of a well-constructed dashboard can significantly impact the troubleshooting process. Key strategies include understanding the metrics being monitored, maintaining simplicity to avoid information overload, starting small and scaling gradually, focusing on readability for team collaboration, and utilizing Grafana's variables feature to manage multiple monitored objects seamlessly. Additionally, leveraging community-contributed dashboards can save time and effort, as Grafana's popularity is bolstered by its extensive community support. While Grafana and Kibana share some best practices, Grafana is often preferred for its support of multiple data sources, making it a top choice for visualizing complex monitoring systems.

Logz.io has introduced Logz.io Reports, a feature designed to automate the generation and distribution of reports based on Kibana dashboards for operational and security purposes. This tool simplifies the process of creating scheduled reports by allowing users to customize report frequency, content, and recipients, with options to deliver via email or Slack. The Reports page under Alerts & Events enables users to manage their reports easily, with options to tweak, disable, or delete them as needed. Users can set up cron schedules for report generation and conduct test runs to ensure accuracy before finalizing. This feature is available across all Logz.io plans, with a limit of 50 active reports per account, and the company encourages user feedback to enhance the service.