July 2017 Summaries

Cloud Foundry is a leading cloud platform as a service (PaaS) that simplifies application development and deployment by managing underlying infrastructure concerns, but requires users to handle application troubleshooting and monitoring. To facilitate log management, Cloud Foundry's Loggregator component collects and aggregates logs from platform components and applications, providing access via command line interface (CLI) or management console. Users can export these logs to external solutions like the ELK Stack (Elasticsearch, Logstash, Kibana) for enhanced analysis and visualization. This process involves deploying a sample application on Cloud Foundry, configuring Logstash to parse logs according to the RFC5424 syslog standard, and creating a log-draining service to forward logs to ELK or Logz.io. Kibana can then be used to create various visualizations to analyze traffic, application actions, and other metrics. While Cloud Foundry's built-in logging services may suffice for smaller applications, larger environments benefit from a centralized logging solution like ELK, which requires careful Logstash configuration for optimal parsing.

Released in conjunction with SysAdmin Day, the second annual Logz.io DevOps Pulse survey highlights the evolving landscape of DevOps, surveying over 700 professionals primarily from North America's tech sector. The survey, conducted in June, consisted of 48 questions covering a range of topics from salaries to software architecture, with a special focus on DevOps culture. Key findings include that while 90% of respondents acknowledge the positive impact of increased collaboration on product quality, 70% express concerns about burnout. Notably, 50% of companies are still in the early stages of implementing DevOps, facing challenges such as modifying company incentives, maintaining communication, and ensuring process flexibility. Despite these challenges, DevOps is credited with shifting team roles and perspectives, though the lack of time and expertise continues to hinder its full implementation. The survey aims to provide insights into the challenges and successes of DevOps practices, offering valuable data to the community for future improvements.

Maintaining cloud production environments requires meticulous monitoring and data collection to prevent and manage service outages effectively, utilizing logs, metrics, or a combination of both. Metrics, which measure component functionality, help DevOps teams assess service value and monitor application performance, while logs provide detailed records of specific events, aiding in incident investigation and root-cause analysis. Though both methods are essential for different aspects of cloud application monitoring, they present challenges in data management and can lead to inefficiencies if improperly implemented. Tracing, a more granular form of monitoring, can be valuable but may overwhelm with data if not carefully managed. Successful integration of these methodologies involves using specialized tools that can extract and analyze significant data, such as the ELK Stack, to streamline the process and enhance application maintenance.

Kibana's Time Series Visual Builder, initially an experimental feature, is set to officially debut in version 6.0, bringing enhanced tools for time series data analysis and visualization. While Kibana traditionally struggled with time series analysis, the Visual Builder offers a comprehensive solution with various visualization types, including Time Series, Metric, Top N, Gauge, and Markdown. Users can customize data and display settings, apply different aggregation types, and utilize advanced calculations like derivatives or moving averages. This new tool positions Kibana more competitively against Grafana, especially in the context of time series visualization, though Kibana remains primarily integrated with Elasticsearch, whereas Grafana supports multiple data stores. The article also touches on the potential impact of the Visual Builder on the future of Timelion, another time series tool within Kibana.

In an exploration of alternative setups for the ELK Stack, PJ Hagerty discusses deploying the stack on a Raspberry Pi instead of using cloud services. The experiment highlights the use of a Raspberry Pi Model B, running a Debian variant called Raspbian Wheezy, to install and configure Elasticsearch, Logstash, and Kibana. The process involves updating system packages, installing a suitable Java runtime, and configuring network settings for each ELK component. Although the guide successfully demonstrates the ELK Stack installation on a Raspberry Pi, Hagerty emphasizes that deploying it in the cloud remains the optimal solution for production environments due to ease of setup and maintenance.

Logz.io announced a contest inviting ELK Stack users to submit their best Kibana dashboards for a chance to win a comprehensive package to AWS re:Invent 2017, including a ticket, flight, hotel, and transportation. Kibana is popular for its ability to create detailed visualizations and dashboards for analyzing log data, with applications across various fields such as application monitoring, business intelligence, and security. To support users, Logz.io provides ELK Apps, a library of premade dashboards and visualizations, which participants can contribute to by submitting their dashboards as part of the contest. The contest criteria are based on the relevance and interest of the use case, the number of visualizations, and the variety of visualization types, with finalists required to write articles about their dashboards for publication. A public vote will determine the winner, and the contest encourages participants to focus on creating dashboards that are broadly useful to the community.

In the article, Daniel Berman explores the challenges and solutions associated with incident management in software development, specifically focusing on how tools like PagerDuty, OpsGenie, VictorOps, Jira Service Desk, and FreshService can aid in effectively managing production incidents. Despite efforts by organizations like NASA to develop high-quality software, defects can still arise, necessitating efficient incident management to minimize downtime and maintain operational standards. Modern DevOps practices require rapid categorization and prioritization of incidents, often through automated services, to address issues promptly, sometimes even within the same day. Tools like PagerDuty offer features such as automation, contextual alert grouping, and pattern-based escalation, while OpsGenie emphasizes alerting and communication with features like on-call scheduling and escalation management. VictorOps provides a real-time platform with customizable alerts and a collaborative chat function, whereas Jira Service Desk offers integration capabilities and a self-service module to streamline service provision. FreshService focuses on ITIL methodologies, enabling automatic incident routing and escalation, and includes a module specifically for major incidents. These tools collectively aim to enhance incident response efficiency, requiring organizations to choose based on their specific goals and standards.