April 2019 Summaries

In April 2019, AWS announced Open Distro for Elasticsearch, sparking discussions about its implications for the ELK Stack and its community. The Open Distro, despite its name, includes not only Elasticsearch but also Kibana and additional plugins, excluding Logstash and Beats. At version 0.8.0, it is not yet production-ready, with installation available via RPM package or Docker images. Notably, the security plugin, based on Search Guard code, is enabled by default, offering features such as role-based access and audit logging. The alerting mechanism allows users to configure alerts for Elasticsearch indices, although it remains somewhat undeveloped. The Performance Analyzer plugin provides metrics monitoring, albeit without time-based analysis. While Open Distro offers Apache 2.0 licensed features for free, maintaining large-scale data pipelines with it or the ELK Stack incurs significant operational costs. Despite these challenges, the Open Distro initiative is seen as a positive step toward greater innovation within the ELK ecosystem.

When deciding between cloud and on-premises solutions for deploying new systems, organizations must weigh security and compliance concerns amidst the evolving landscape of shared versus dedicated environments. Cloud providers have advanced their security features and compliance certifications, but some organizations prefer on-prem solutions due to perceived control and security benefits. Evaluating on-prem solutions involves assessing internal security controls and technical requirements, while cloud solutions offer flexible, scalable services that can help meet specific compliance needs. Ultimately, the decision should be informed by a careful assessment of the technical and security requirements, with an understanding that both cloud and on-premises options have unique advantages and challenges. Organizations must prioritize strong project management and ensure compliance with industry standards, regardless of the chosen deployment method, to maintain security and operational efficiency.

The blog post explores the evolution from monolithic software to microservices and nanoservices, highlighting the advantages and challenges of each approach. Monolithic applications, often cumbersome and difficult to maintain, led to the creation of microservices, which decompose applications into smaller, independent units that align with business processes, thereby enhancing scalability and maintainability. Microservices integrate well with DevOps practices but can still suffer from complexity and dependency issues. Nanoservices, an even more granular approach, promise greater modularity and flexibility by enabling services to be independently developed and deployed. However, they introduce new challenges in monitoring and managing the increased number of services, potentially creating a complex system that is difficult to control. The article concludes that while microservices are a mature, reliable choice for larger organizations, nanoservices offer promising innovations for those willing to experiment with cutting-edge technologies, despite their current limitations and potential for increased complexity.

The blog post by Daniel Berman provides a detailed guide on installing the ELK Stack, an open-source log aggregation and analytics platform comprising Elasticsearch, Logstash, and Kibana, on Amazon Web Services (AWS). It outlines the process of setting up a sandbox environment using a single AWS Ubuntu instance, detailing the installation and configuration steps for each component: Elasticsearch as the NoSQL database and search server, Logstash for log shipping and parsing, and Kibana for data visualization. The tutorial emphasizes the importance of security and configuration adjustments required for production environments, such as using multiple EC2 instances and securing ports. It also covers the creation and definition of Elasticsearch index patterns in Kibana, allowing users to visualize and analyze log data effectively. The post highlights that while setting up the ELK Stack can simplify log management and analysis, careful consideration is needed for production deployments to ensure performance and security.

In an imaginative use of their machine data analytics platform, Logz.io analyzed Twitter data to gauge the popularity of Game of Thrones characters and their potential to claim the iron throne, coinciding with the airing of the show's final season. By connecting Twitter's API to their log analysis platform, they tracked hashtags related to prominent characters and correlated them with mentions of the "iron throne." The data revealed Daenerys Targaryen as the most popular character with 21% of mentions, closely followed by Arya Stark at 17.5% and Cersei at 12%. Daenerys also emerged as the top contender for the iron throne with 20% of the votes, followed by Jon Snow and Tyrion Lannister. The analysis highlighted how trends and theories among fans evolved over time, underscoring the utility of log analytics tools like Kibana to visualize data trends across diverse contexts beyond traditional IT applications.

In a rapidly evolving threat landscape, maintaining the security of enterprise networks is critical, with open-source network-based intrusion detection systems (NIDS) playing a vital role in identifying and mitigating threats. The text discusses five notable NIDS: Snort, Suricata, Zeek, OpenWIPS-ng, and Sguil, each offering unique capabilities and benefits. Snort, maintained by Cisco Systems, is notable for its community-driven rule base and versatile detection modes, while Suricata is praised for its real-time capabilities and multithreading. Zeek provides deep network monitoring and operates on the application layer, offering comprehensive protocol analysis. OpenWIPS-ng is specialized for wireless networks, developed by the Aircrack-ng team, and Sguil focuses on efficient data presentation and alert management. These systems support both signature-based and anomaly-based detection methods, though each has its own limitations, such as resource demands or deployment complexity. By understanding these systems' strengths and weaknesses, organizations can better protect their networks from intrusions and data theft.

Daniel Berman's blog post discusses the use of the ELK Stack—Elasticsearch, Logstash (or Fluentd), and Kibana—as a preferred tool for Kubernetes users seeking effective logging solutions, emphasizing the growing preference for Fluentd, thus forming the EFK Stack. The article provides a step-by-step guide on deploying the Elastic GKE Logging app, a turnkey solution for setting up a logging infrastructure on Google Kubernetes Engine (GKE) using Elasticsearch, Kibana, and Fluentd. Key steps include preparing the GKE environment, deploying the EFK Stack, reviewing deployment configurations, and accessing Kibana for log analysis. Berman highlights that while the Elastic GKE Logging app is suitable for development or quick setups, more advanced deployment methods are recommended for production environments to optimize performance and access. The post concludes by suggesting improvements like using ingress for external access and employing Curator for managing Elasticsearch clusters.