March 2018 Summaries

This blog post by Ido Halevi outlines a method for shipping and analyzing AWS Cost and Usage reports using AWS Lambda and Logz.io’s ELK Stack. It provides a step-by-step guide on setting up a pipeline with a Lambda shipper to process the reports, which are CSV files delivered to an S3 bucket and updated up to three times daily. These reports help track AWS service usage and costs, though their size can make them challenging to analyze. The post details the process of enabling reports in an AWS account, including creating an S3 bucket, setting permissions, and configuring report settings. It also describes creating a Lambda function to ship data to Logz.io, with specific instructions for setting triggers and permissions, aiming to facilitate cost optimization and monitoring through Kibana queries and visualizations.

In the context of cloud computing, the rise of containers has been significant, with Docker popularizing the concept initially introduced by older technologies like BSD Jails and Solaris Zones. Amazon Web Services (AWS) quickly entered the market with its Elastic Container Service (ECS) in 2014, followed by the introduction of Elastic Kubernetes Service (EKS) and the Fargate launch type, which aims to simplify container management by removing the need for infrastructure management. Fargate, in particular, is seen as a game-changer, allowing users to focus on developing services without worrying about the complexities of provisioning and managing containerized environments. Despite being the last to enter the managed Kubernetes market, AWS is poised to compete strongly against Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS), with Fargate providing a unique competitive advantage. Kubernetes has become a crucial technology for cloud providers, and the primary challenge now lies in choosing the right vendor and approach for implementation.

Kibana offers users extensive flexibility in visualizing data indexed in Elasticsearch, but some tasks can be complex due to either a lack of experience or missing features. To address these challenges, various workarounds are suggested, including embedding images using markdown visualization and base64 encoding, hyperlinking fields by changing their format to URL, and adding log messages to dashboards using saved searches. Formatting fields can enhance readability by adjusting numeral.js format patterns, while Kibana’s look and feel can be customized by editing Less stylesheets for users with JavaScript and CSS skills. These strategies aim to improve the functionality and user experience of Kibana dashboards, offering creative solutions to common limitations.

Logz.io, a rapidly expanding SaaS company, has been recognized among the top 100 fastest growing SaaS organizations globally in the first quarter of 2018, as per the SaaS 1000 list. This acknowledgment is attributed to its ambitious growth strategies and innovative product offerings, including AI-based Application Insights, Data Optimizer, Parsing, and Live Tail, alongside open-source tools like Sawmill and Apollo. Following a successful $23 million Series C funding in November 2017, Logz.io has expanded its workforce in Boston and Tel Aviv, highlighting its commitment to enhancing its scalable and intelligent log analysis technology. The company's growth is indicative of its success in meeting significant needs within the IT operations industry, empowering organizations to derive maximum insights from machine data and enhancing the efficiency of the log analysis process. Logz.io's achievements and continuous innovation reflect its dedication to scaling ELK and supporting enterprises in leveraging this powerful open-source tool, while also preparing for further growth in both size and service offerings.

The blog post explores the integration and utilization of Bro, an open-source network intrusion detection system (NIDS), with the ELK Stack for analyzing network traffic logs, specifically focusing on Bro connection logs. It emphasizes setting up a Kibana dashboard to visualize various network metrics such as connection states, protocols, and geographic origin of connections. The data from Bro logs can be enriched and visualized using Kibana's diverse tools like Coordinate Maps, Pie Charts, and Line Charts for identifying potential network threats. The article also highlights the importance of implementing an alerting mechanism to detect abnormal network activities, suggesting the use of Bro's notification framework or Logz.io's built-in alerting engine for real-time notifications. Ultimately, integrating Bro with a centralized logging platform like the ELK Stack enhances data analysis and visualization capabilities, providing deeper insights into network activities.

In the blog post, Lauren Sirt emphasizes the significance of International Women's Day, particularly in the tech industry, by highlighting the contributions and experiences of two female DevOps professionals, Shahar Gotshtat and Frida Gorovoy, at Logz.io. Shahar's journey began with a natural aptitude for STEM, leading her to a career in DevOps, where she appreciates the holistic view of systems and the need for a broad understanding across various platforms. She advocates for challenging gender stereotypes and fostering a parent-friendly workspace to retain female talent. Frida's interest in DevOps stemmed from her early passion for gaming and her experience in the military's IT unit, where she developed a love for learning and problem-solving within changing and dynamic environments. She discusses the importance of diverse perspectives and the need for women to be genuinely interested in tech rather than feeling compelled to enter the field due to gender disparities. Both women highlight the necessity of encouraging more female role models and breaking down cultural barriers that deter women from pursuing STEM careers.

Infrastructure as Code (IAC) is a transformative approach that treats manual configurations and procedures as code, automating and ensuring consistency across infrastructure deployments. Tools like Terraform, Ansible, and Puppet each offer unique strengths for managing IAC, addressing issues like configuration drift and human error. Terraform, with its cloud-agnostic and efficient provisioning capabilities, excels in managing cloud infrastructure but struggles with managing state files. Ansible is favored for its ease of use and rapid deployment, although it lacks continuous state management. Puppet, the most mature of the three, offers robust state management and appeals to developers due to its software development-like paradigm, but its complexity can be a barrier to entry. Each tool is backed by strong community support, providing resources and innovations that enhance their functionalities, with the choice of tool often depending on specific organizational needs and team expertise.