October 2017 Summaries

Sisense, a Business Intelligence data analytics solution, employs Logz.io’s ELK-based platform to monitor and log various server and client-side services, facilitating troubleshooting and performance optimization. Central to this process is the Sisense Monitor, an embedded Kibana dashboard that allows users to access and analyze log data independently. By leveraging different types of log data, users can visualize server performance, query metrics, and data collection processes via interactive visualizations such as pie charts and Gantt charts. This self-service tool reduces dependency on the Customer Support team, enhancing user autonomy and platform supportability by integrating Sisense with Logz.io for improved data analysis and observability.

Elasticsearch's extensive REST API is a fundamental feature that allows users to integrate, manage, and query indexed data in diverse ways, with numerous examples of its application across various industries. The article provides an overview of essential API categories such as Document, Search, Indices, cat, and Cluster APIs, each serving distinct purposes like handling documents, querying data, managing indices, formatting data outputs, and managing clusters. While this guide highlights key API calls and offers practical cURL command examples, it is not exhaustive, and advanced users might benefit from an accompanying cheat sheet on Elasticsearch Cluster API best practices. The REST API's versatility is a significant factor in the popularity of Elasticsearch and the ELK stack, and the article encourages experimentation to gain proficiency. Additionally, it notes that APIs may evolve over time, necessitating awareness of updates and deprecations to maintain effective usage. For users of the managed ELK solution Logz.io, a public API based on the Elasticsearch Search API is available, albeit with some limitations, allowing for search queries and alert management.

Selecting a storage solution for time series data involves considering various factors, with InfluxDB and Elasticsearch being two prominent options. Although InfluxDB is specifically designed for time series data, offering efficient handling of high-frequency write requests and optimized data storage with its LSM tree paradigm, Elasticsearch, originally intended for document indexing, is often used for time series data due to its robust search and aggregation capabilities. InfluxDB excels in managing numerical data with its built-in aggregation functions and SQL-like interface, while Elasticsearch offers superior text search abilities and an extendable aggregation framework, making it suitable for applications that require analyzing both numerical and textual data. Furthermore, Elasticsearch's integration with the ELK Stack components like Logstash and Kibana enhances its utility in log aggregation and monitoring scenarios. The choice between these databases largely depends on the specific data requirements, such as the need for handling textual data or the frequency of data writes, with InfluxDB being more specialized for time series-centric applications and Elasticsearch providing broader search functionalities.

Centralized logging has become vital for implementing security in cloud environments, particularly for organizations using AWS services, as it allows for comprehensive log management and analysis crucial to security strategies. AWS provides a variety of tools that generate log data, which can be aggregated to give a centralized security overview, but setting up this system requires understanding how to access, extract, store, and secure this data. The ELK Stack, consisting of Elasticsearch, Logstash, and Kibana, is widely used on AWS to manage and visualize log data, aiding in building a Security Information and Event Management (SIEM) system by aggregating, processing, and analyzing logs from various AWS services like VPC Flow Logs, CloudTrail Logs, and ELB Access Logs. However, ELK lacks built-in alerting and retention capabilities, necessitating additional tools or services to handle these aspects, and the sheer volume of log data requires sophisticated analysis tools to uncover security insights. Recent security breaches, such as the leak of sensitive information from AWS S3, underscore the importance of implementing robust security policies and centralized logging strategies to prevent vulnerabilities and maintain cloud security.

Logz.io has introduced a new Audit Trail feature to its hosted ELK-as-a-Service platform, enhancing administrators' ability to monitor user activity for both internal management and compliance with external certification programs. This feature logs various user actions, including logging in, creating alerts, and updating dashboards, and presents them in a user-friendly table accessible via the Logz.io UI's Settings page. The table can be filtered by user ID, date and time, and server IP, with options to customize columns and sort data. Additionally, the Audit Trail information can be exported as a CSV file for easy sharing, providing users with a comprehensive tool for tracking and managing user activity within their Logz.io accounts.

Filebeat, a reliable log shipper for ELK-based logging architectures, has evolved from its predecessors Lumberjack and Logstash-Forwarder to become an essential tool, though not without its challenges. Users need to be cautious with YAML syntax, as its sensitivity to indentation can lead to configuration errors. The Filebeat registry file, which tracks the state of log files, can grow large and consume memory, especially when many new log files are created, but options like clean_removed and clean_inactive can mitigate this issue. File handlers for removed or renamed log files can also consume resources, and configurations such as close_inactive and closed_removed can help manage these handlers. When configuring multiple pipelines, Filebeat allows for tracking multiple log files by defining file paths in inputs, but specific settings for each log type should be added to avoid configuration pitfalls. Although rare, high CPU usage can occur and is influenced by the scan_frequency setting, which can be adjusted to manage resource consumption. Despite these challenges, Filebeat's default settings are usually sufficient, and tools like the Filebeat wizard developed by Logz.io can help users avoid common configuration issues.

With cybersecurity breaches on the rise, organizations are under increasing pressure to safeguard their data, leading to the adoption of stringent security regulations like the EU's GDPR and the U.S.'s HIPAA. Logz.io has responded by achieving multiple security certifications, including SOC-2 Type 2 and HIPAA, to demonstrate their commitment to high security standards and data protection. Their secure log analysis platform helps SecOps teams proactively identify threats and potential breaches, though it requires trusting a third party with sensitive data. To mitigate this risk, Logz.io has implemented robust security measures to ensure customer data remains protected, earning trust from clients across various industries, including those with specific regulatory needs. Their continuous efforts in developing technologies and gaining certifications further reinforce their dedication to data security.

The Logz.io Kibana dashboard contest, announced in July 2017, showcased the visualization capabilities of Kibana across various use cases, with finalists evaluated based on the relevance and impact of their dashboards, the number and types of visualizations used. Among the notable submissions were an AWS security dashboard providing comprehensive monitoring of data from various AWS services, a MongoDB dashboard offering detailed insights into database queries and transactions, an Apache access log dashboard for monitoring API requests, and a Docker dashboard featuring a mix of visualizations based on container logs and performance metrics. The contest aimed to highlight innovative uses of the ELK Stack, with participants utilizing Logz.io's built-in support and automatic parsing features to ship and analyze logs. The winning submission was to be awarded a package to AWS re:Invent 2017, with the community invited to vote on the finalists, ensuring an unbiased selection process by omitting personal information from the submissions.