March 2020 Summaries

In the blog post by Dotan Horovits, the focus is on securing Elasticsearch clusters to prevent data breaches, emphasizing the importance of not connecting Elasticsearch directly to the internet and employing encryption for data at rest. The author suggests implementing robust authentication measures, such as role-based access control and various user authentication methods, while also recommending regular software updates to address vulnerabilities. Backing up data is highlighted as a crucial step to ensure recovery in case of breaches, with the Snapshot API for Amazon S3 buckets suggested as a backup method. Horovits notes that most breaches are due to inadequate adherence to security best practices rather than intrinsic vulnerabilities in Elasticsearch itself. The post concludes by advocating for managed services like Logz.io to simplify and enhance security compliance, reducing the operational burden on organizations.

In response to the COVID-19 pandemic, Logz.io has launched a Community COVID-19 Dashboard Project to provide clear and accessible data about the virus's impact globally and regionally. This initiative invites engineers and data scientists to contribute data, not only on public health but also on other societal aspects, using open-source software and crowdsourced information. The dashboard currently utilizes data from the John Hopkins University CSSE, offering insights into confirmed cases and deaths worldwide, with detailed views on specific countries and states. The project aims to expand its data sources to include health-related information and broader impacts on the economy, air travel, environment, and internet usage, creating a comprehensive virtual data hub for professionals and the public. This effort represents a collaborative attempt to offer clarity and foster problem-solving during the pandemic.

Logz.io, recognizing the economic challenges posed by the COVID-19 pandemic, has initiated a Suppliers Support Plan to aid small businesses, which they regard as crucial to the local economy. The company intends to purchase services upfront from suppliers who meet certain criteria, such as having 25 or fewer employees and having engaged with Logz.io within the past three months, to help them navigate these difficult times. The plan is open to current suppliers of Logz.io, and applications are assessed at the company's discretion, with payments being made upon mutual agreement and against a valid tax invoice. This initiative reflects Logz.io's commitment to supporting its vendors during a period of economic uncertainty.

Security Information and Event Management (SIEM) systems are crucial for modern organizations, gathering and analyzing events from various systems to detect and manage security threats. The blog explores the trade-offs between building a custom SIEM using free and open-source software (FOSS) versus adopting commercial solutions, highlighting that while FOSS may appear cost-effective, it involves significant operational costs and maintenance efforts. The evolution of SIEMs, driven by advances in cloud technology and AI, has made commercial solutions more affordable, often adopting flexible pay-as-you-go models. These solutions can alleviate infrastructure maintenance burdens, offering significant cost management benefits. However, choosing between a DIY or managed SIEM solution requires careful consideration of both capital (CAPEX) and operational expenditures (OPEX), as well as the technical capabilities necessary to meet an organization's security and compliance needs. Ultimately, organizations must weigh the hidden costs and operational demands of FOSS against the benefits of cloud-based SIEM solutions, which offer scalability and ease of use, to determine the best fit for their security strategy.

In a modern landscape dominated by microservices and containerized applications, Kubernetes has emerged as a leading orchestrator for managing complex distributed systems. With this complexity comes the need for robust observability, which relies on the three pillars of logs, metrics, and traces to debug and maintain systems efficiently. The article explores how these pillars enhance system observability in Kubernetes environments, with logs capturing discrete system events, metrics providing numerical insights into system health, and traces revealing the end-to-end request paths within distributed systems. Various open-source tools are highlighted for implementing Kubernetes observability, including Jaeger for distributed tracing, Fluentd for log collection, Prometheus for metric automation, Grafana for data visualization, and OpenSearch for log analysis. Additionally, OpenTelemetry is introduced as a comprehensive framework for capturing observability signals in a cloud-native, vendor-agnostic manner. The Logz.io Open 360 platform is presented as a unified solution that integrates these open-source tools to offer a comprehensive view of Kubernetes environments, enhancing the ability to monitor application health, debug issues, and understand the impacts of changes within the infrastructure.

Monitoring in the computing world has evolved significantly with the advent of cloud-native solutions like Kubernetes, which present new challenges due to their dynamic nature. Traditional monitoring systems often struggle with such environments, prompting a comparison between Prometheus and InfluxDB as two modern solutions. Prometheus, optimized for time-series data, is integrated with the Cloud Native Computing Foundation (CNCF) and utilizes a pull model for scalability, featuring a flexible query language called PromQL. InfluxDB, also a time-series database, excels in high-availability storage and supports both push and pull data collection methods, making it suitable for diverse applications beyond monitoring, such as IoT and sensor data. While Prometheus is favored for cloud-native monitoring due to its numerous integrations and community support, InfluxDB is preferred for broader use cases involving time-series data. Both systems benefit from enhanced visualization and dashboard capabilities when used alongside tools like OSS Grafana, and Logz.io offers a managed service that integrates Prometheus metrics with log and trace analytics for a comprehensive observability platform.

In response to the surge in remote work due to COVID-19, Logz.io launched the Observability All-Star program to recognize engineers who excel in leveraging machine data for comprehensive visibility into production environments. These Observability All-Stars are celebrated for their ability to maintain high-performing infrastructure and applications by effectively using the right tools at scale, correlating metrics and logs, and swiftly resolving issues. The program invites nominations for individuals who demonstrate exceptional use of cloud observability solutions in monitoring, troubleshooting, and securing production issues. All-Star nominees are reviewed by Logz.io's Product and Marketing teams, and winners are announced monthly, receiving recognition through various channels including the Logz.io blog and social media. The initiative aims to support and celebrate the achievements of engineers dedicated to maintaining service reliability during challenging times.

Logz.io announced its 2020 Cloud Observability Webinar Series, which aims to provide knowledge on DevOps, open source, and observability amidst the challenges of attending in-person events due to COVID-19. The series, running every Wednesday until the end of June, features webinars on topics such as Kubernetes monitoring, Kibana updates, and metrics monitoring for AWS and Azure. Presentations will be delivered by Logz.io's Product Managers, Engineers, Customer Success Engineers, and partners. Highlights include sessions on resolving Kubernetes production issues, Azure observability, and infrastructure monitoring. Interested participants can register to stay updated on upcoming webinars and enjoy a free 14-day trial of Logz.io's AI-powered observability tools.

Establishing an effective monitoring solution for software systems involves several steps, including deciding what to monitor, choosing collection and storage methods, and selecting visualization tools. Grafana, with its pluggable backends and built-in alerting, is recommended for visualizing collected metrics from various monitoring solutions, such as InfluxDB, a time series database used for high-availability storage and rapid retrieval of time series data. The text details the process of deploying InfluxDB and Grafana within a Kubernetes cluster using Helm charts, as well as configuring Grafana to display data from InfluxDB. It also covers how to set up Heapster for feeding data into InfluxDB, creating Grafana dashboards, and configuring alerts. The article suggests exploring additional technologies like Telegraf, Chronograf, and Kapacitor, which form the TICK stack, to enhance InfluxDB's capabilities, and emphasizes the importance of monitoring systems to anticipate and respond to unexpected application behaviors.

Amidst the global challenges posed by the COVID-19 pandemic, Logz.io emphasizes the critical role of its technology in supporting healthcare, financial, and critical infrastructure organizations worldwide, ensuring their services remain operational at the highest levels. The company has implemented measures to safeguard its employees, including halting international travel, minimizing face-to-face meetings, and allowing remote work, while adhering to local authority guidelines across its operational countries. Logz.io's multi-cloud, multi-region SaaS operation, running on AWS and Azure across six regions, is designed for resilience and availability, with disaster recovery plans enabling continuity even if a cloud provider fails. Additionally, to aid in the fight against COVID-19, Logz.io offers free access to its service for health organizations working on treatments and research, inviting these entities to share their efforts to receive support.

Stockholm has emerged as a key hub for innovation in Europe, with a thriving startup ecosystem that attracts developers from Scandinavia and the Baltics, and is home to notable companies like Klarna and iZettle, as well as tech giants like Microsoft and Google. This vibrant environment supports a variety of meetups focusing on the latest trends in DevOps, covering topics such as Kubernetes, OpenStack, AWS, Agile methodologies, open-source software, and more. The city hosts numerous events catering to different interests, with meetups organized by passionate groups that facilitate networking and knowledge sharing among developers and tech enthusiasts. These gatherings vary in size and frequency, offering platforms for discussing cutting-edge topics like chaos engineering, serverless architectures, and database management, thereby contributing to Stockholm's reputation as a dynamic center for tech innovation.

YAML, an acronym for "Ain't Markup Language," is a readable data serialization language widely used in configuration files for software systems, particularly within DevOps environments. It is a superset of JSON, offering more legibility and features like lists, maps, and scalar types, making it preferable for configuration tasks over other markup languages like XML. YAML’s importance is highlighted in systems like Kubernetes, where its declarative nature simplifies complex configurations and scaling. In the context of the ELK Stack, YAML files (.yml or .yaml) are crucial for configuring components like Elasticsearch, Kibana, Logstash, and Beats, allowing users to set parameters such as ports, paths, and SSL configurations. The document provides detailed instructions on configuring these components through YAML, emphasizing the importance of uncommenting lines to activate configurations. Advanced settings for Elasticsearch clusters and Kibana involve specifying memory settings and SSL certificates, while Logstash and Filebeat configurations require attention to JSON and YAML files for defining inputs, filters, outputs, and modules. The use of YAML in these contexts demonstrates its utility in managing configuration complexity and enhancing operational efficiency.

Threat intelligence feeds play a vital role in modern cybersecurity by tracking IP addresses and URLs linked to various online threats such as phishing, malware, and ransomware. Despite the abundance of open source threat intelligence feeds, their reliability and detail vary, with some of the best not being frequently updated. The article highlights several noteworthy feeds, including Emerging Threats by Proofpoint, which provides categorized and confidence-scored data on malicious activities, and FBI InfraGard, a collaboration between the FBI and the private sector focusing on critical infrastructure threats. Other notable feeds include Dan.me.uk's tools for IP and domain information, the CINS Score by Sentinel, which rates IP addresses' trustworthiness, and Blocklist.de, which tracks server attacks. It also mentions hpHosts, a community-managed database of malicious IPs, and AlienVault OTX, which uses community-generated and automated pulses to rank IPs by reputation. Additionally, abuse.ch offers specialized feeds like Feodo Tracker for botnets and URLhaus for malicious domain data, both providing comprehensive and regularly updated datasets accessible via APIs.