August 2017 Summaries

Elasticsearch 6.0.0-beta2, based on Lucene 7-SNAPSHOT, has been released as the fourth pre-6.0.0 version, aimed at allowing users to test features and provide feedback before the official release. This beta version is intended solely for testing purposes and should not be used in production, as indices created with it will not be compatible with the forthcoming 6.0.0 GA version, and upgrading from 6.0.0-beta2 to any other version is not supported. A notable improvement in this release is the enhanced security bootstrap experience, where default passwords have been removed in X-Pack Security, and a random seed is now auto-generated for the keystore, reducing user interaction unless other secrets are to be stored. Users are encouraged to try this beta version and share their feedback via Twitter, forums, or the GitHub issues page.

Elastic is committed to inclusivity and diversity, which is reflected in its efforts to make Kibana, its data visualization tool, accessible to all users. With the release of Kibana 6.0, the company has initiated improvements in color contrast, keyboard accessibility, and support for assistive technologies like screen readers, guided by the Web Content Accessibility Guidelines (WCAG). These changes include removing the low-contrast pink from the side navigation and enhancing keyboard navigation to make the interface more user-friendly. Elastic acknowledges past oversights in accessibility and invites users to contribute feedback and suggestions to further develop Kibana’s accessibility features through its Pioneer Program. The company's CEO, Shay Banon, emphasizes the alignment of product development with Elastic’s core values of open-mindedness and inclusivity, encouraging a culture where diverse perspectives are integral to its operations.

Collector Bank, a leading digital niche bank in the Nordics, has successfully transitioned its technological infrastructure to the Microsoft Azure Cloud, leveraging Elasticsearch as a pivotal tool for innovation and efficiency. Over the past three years, Elasticsearch has transformed the bank’s logging and data visualization processes, enabling rapid development and improved service quality. Initially reliant on various systems for payment solutions and loans, the bank integrated Elasticsearch to enhance log searchability and system correlation, fostering a more effective workflow. Today, with 18 development teams, Collector Bank employs Kibana dashboards for real-time monitoring and alerts, supported by devops tools like PagerDuty and Slack. The architecture involves Azure Web Apps and Event Hubs, utilizing C# and Serilog for logging, and addressing challenges such as data type collisions with a suffix strategy. Looking ahead, the bank plans to deploy Elastic Cloud Enterprise for enhanced cluster management and security, and explore machine learning for fraud detection and a digital document archive, ensuring compliance with upcoming GDPR requirements.

The Ingest to Logstash config migration tool introduced in Logstash 6.0 simplifies the process of converting Ingest node configurations into equivalent Logstash configurations, significantly reducing the manual effort involved in such transitions. This tool is especially valuable for handling complex transformations, such as converting field names and understanding Logstash-specific syntax, by enabling automatic conversion of Ingest filter configurations, saving users considerable time and effort. Implemented in JavaScript and executable via Java's Nashorn scripting engine, the tool can be run using a wrapper script, making it portable and easy to use. It also supports quick configuration setup by appending standard input and output options, facilitating rapid prototyping of Logstash environments. The tool is poised for integration into Logstash's future user interface, offering a more streamlined approach to managing configuration conversions, and users are encouraged to provide feedback to further improve its functionality.

The text discusses the importance of data visualization in the process of building and validating machine learning models, specifically within the context of developing a malware scoring model called MalwareScoreTM at Endgame. Phil Roth created an internal tool named Bit Inspector to facilitate model validation through visualizations, which proved essential in bridging communication between data scientists, engineers, and managers. Initially intended for developers, Bit Inspector's audience expanded to include subject matter experts and executives, necessitating a focus on explainability to cater to diverse users. The text highlights the evolving role of data visualization in enhancing understanding and trust in machine learning processes and underscores its growing acceptance within the information security industry, as evidenced by events like BSidesLV.

Elasticsearch 6.0 introduces a new feature called Index Sorting, which optimizes the storage of documents on disk by ordering them in a user-specified manner. This enhancement, originating from Lucene's IndexSorter tool, allows for significant improvements in query performance through a mechanism known as "early termination," where queries can be short-circuited after retrieving a specified number of sorted documents. Historically, Lucene's static sorting tool was replaced with a dynamic merge policy that allows sorting during the merge process, but Elasticsearch now allows sorting at flush time to enhance efficiency further. While this feature can greatly benefit search response times, especially with large datasets, it comes with a trade-off of reduced indexing performance due to the additional overhead of sorting. The decision to implement Index Sorting should, therefore, carefully consider the balance between write and query performance. Practical examples include creating efficient leaderboard queries for gaming applications by sorting documents based on player scores. However, if optimized for write performance, this feature might not be suitable due to its potential to decrease write throughput by up to 50%. Testing with specific use cases and datasets is crucial to determine its effectiveness.

Elastic announced the beta release of Elastic Cloud on Google Cloud Platform (GCP) on August 17, 2017, expanding their hosting options for Elasticsearch and Kibana beyond Amazon Web Services (AWS). This move aligns with Elastic's goal to provide their services wherever users are, and the partnership with GCP offers users flexibility in managing their projects by bringing Elastic products closer to applications on GCP. The beta version allows users to access the latest open-source software and X-Pack features, with support available under a standard policy, but without service level agreements (SLAs) until the general availability (GA) release scheduled for later in 2017. Users can create clusters on GCP using existing Elastic Cloud accounts, and the beta release encourages feedback to improve the service. The Elastic Stack, enhanced by X-Pack, addresses global use cases in search, logging, metrics, and analytics, and the expansion to GCP reflects a response to community demand and the innovative nature of the platform.

The blog post discusses the integration of Wireshark, Elasticsearch, and Kibana for network packet analysis, emphasizing the importance of packet capture and analysis for network administrators and security analysts. Wireshark, a widely used open-source packet capture tool, can output data in JSON format compatible with Elasticsearch, allowing detailed data exploration and visualization in Kibana. The process involves capturing network traffic, parsing protocol fields, and using tools like Packetbeat for indexing data into Elasticsearch, which benefits from a defined mapping to optimize search and aggregation performance. The post highlights the use of Filebeat and Logstash for data processing and transformation, enabling efficient data ingestion into Elasticsearch and thus facilitating comprehensive analysis and visualization in Kibana. The integration supports scalable, real-time exploration of network packets, providing significant insights for IT and security professionals.

Auditbeat, a new addition to the Beats family introduced in version 6.0, is a tool aimed at auditing user and process activities on systems, currently in its beta stage with plans for future enhancements. It offers two primary monitoring capabilities: Linux audit framework monitoring and file integrity monitoring. The tool receives events from the Linux kernel's audit framework and sends them to Elasticsearch, automatically grouping and structuring messages for easier ingestion. For file integrity monitoring, Auditbeat can track changes to specified files or directories across Linux, macOS, and Windows, sending real-time events to Elasticsearch with metadata and cryptographic hashes. This functionality allows users to ensure compliance and identify potential malware. Users are encouraged to try Auditbeat and provide feedback as part of Elastic's Pioneer program.

Elasticsearch 5.5.2, based on Lucene 6.6.0, has been released as the latest stable version and is available for deployment on Elastic Cloud, the Elasticsearch-as-a-service platform. Users are encouraged to download the new version, explore its features, and provide feedback via Twitter or the forum. Any issues encountered can be reported on the GitHub issues page. Additionally, release notes and information about breaking changes for Elasticsearch 5.5 and X-Pack 5.5.2 are accessible to users.

Logstash, a central component for managing data flows between producers and consumers, historically supported only a single pipeline per instance, leading to complex configurations when handling multiple event streams. Users often resorted to conditional logic to separate these streams, resulting in cumbersome and error-prone setups that struggled with issues such as congestion and differing flow requirements. The introduction of the Multiple Pipelines feature in Logstash 6.0 addresses these challenges by allowing a single Logstash instance to manage several distinct pipelines, each defined in a configuration file called `pipelines.yml`. This advancement simplifies configuration management, improves isolation between flows, and supports dynamic reloading of individual pipelines without interrupting others. The feature is designed to enhance user experience by providing flexibility and efficiency, and it is supported by monitoring tools like X-Pack. Users are encouraged to test this feature in its preview release and provide feedback to further improve its functionality.

Elasticsearch's journey to implement sequence numbers in index operations has been complex yet transformative, significantly improving the system's replication and recovery processes. Initially sparked by a "what if" scenario, the idea was to enable features like a Changes API, incremental reindexing, and cross-datacenter replication, but the implementation turned out to be more challenging than expected. After nearly six years and significant efforts, including a complete rewrite of replication logic and collaboration with distributed systems experts, Elasticsearch introduced primary terms and sequence numbers to resolve operational divergences upon primary shard failures. These enhancements allow Elasticsearch to efficiently manage index operation history, enabling faster recovery by using transaction logs instead of large file copies and setting the stage for cross-datacenter replication. The introduction of configurable translog retention settings in version 6.0 underscores the balance between faster recovery and disk utilization. While the deployment of sequence numbers marks a major step forward, the community continues to explore additional features and improvements inspired by this advancement.

Timelion is a powerful visualization tool within Kibana designed for analyzing time series data using a unique syntax-driven approach, allowing users to chain functions to create complex graphs. Unlike traditional visual editors, Timelion's syntax enables users to draw data from different indices or sources into a single graph, offering capabilities that classical Kibana visualizations may not support. Users can manipulate data through various functions like .es for Elasticsearch queries, .wbi for World Bank data, and styling functions for customizing graph appearances. Timelion supports mathematical operations, conditional selections, and function chaining to modify data series, offering advanced features like trend lines, multiple time series comparisons, and different scaling options. While Timelion provides extensive capabilities for time series visualization, Kibana's newer Time Series Visual Builder offers a graphical alternative with overlapping functionalities, providing users with options to choose based on their specific requirements.

Kibana's search experience is being enhanced with the introduction of Kuery, a new query language designed to address limitations of the existing query bar by providing features such as autocomplete, scripted field support, and improved user guidance. Unlike the previous method, where queries were passed directly to Elasticsearch with minimal processing, Kuery aims to make queries more intuitive and powerful by treating them as named functions, which simplifies syntax while allowing for future extensibility through custom query types. Additionally, Kuery seeks to unify and streamline the user interface by removing the separate filter bar and integrating interactive filter capabilities directly into the query bar, making it easier for users to construct and modify queries. This development not only makes the search experience more user-friendly but also sets the stage for future enhancements, inviting user feedback to shape the ongoing evolution of Kibana's query capabilities.

Elasticsearch 6.0.0-beta1, based on Lucene 7-SNAPSHOT, has been released for testing purposes, introducing significant updates aimed at improving performance and security. Key changes include a new Sequence Numbers infrastructure for faster recovery of active indices by allowing replicas to sync with the primary node more efficiently, and a search scalability feature that optimizes shard-level requests to prevent system overloads during large-scale searches. The release also enhances disk space management with a new disk.watermark.flood_stage setting that restricts writes to read-only mode when disk space is critically low, and eliminates default passwords by introducing a bootstrap.password setting to enhance security. Additional improvements include reduced overhead in profiling and search timeouts, better percolator performance through range fields, and updates to package configurations for easier setup. Users are encouraged to test this beta version, although it is not intended for production use, and to provide feedback through designated channels.

Scaling Elasticsearch and evolving the Elastic Stack to fit various needs involves a strategic approach to data collection, storage, and visualization, with a focus on modularity, flexibility, and simplicity. The process begins with minimal configurations using tools like Filebeat and Winlogbeat for log and metric data collection, which can be expanded with Elasticsearch ingest node processors and additional Beat modules like Packetbeat, Metricbeat, and Heartbeat. Introducing Logstash allows for advanced data transformations and handling diverse inputs, while built-in resiliency mechanisms in Filebeat and Logstash ensure data durability and prevent overloading. To address limitations in Logstash's persistent queue, message brokers like Kafka can be integrated to decouple indexing from data collection. As the system grows, different deployment topologies, such as centralized Logstash or multi-data center setups, can be employed to manage configurations and ensure high availability. Kibana, as the primary tool for data visualization, can be scaled with multiple instances and coordinated across Elasticsearch nodes, with Elastic's X-Pack security providing robust access controls. For a more streamlined and resource-isolated approach, Elastic Cloud Enterprise offers a managed service to deploy and manage Elastic Stack instances with added features like versioning, upgrading, and containerization.

Elastic's machine learning capabilities are increasingly being utilized by organizations for various analytics projects, including security and operational analytics, with a focus on anomaly detection in time series data. The process of effectively sizing hardware and clusters for machine learning applications in Elasticsearch requires a prior understanding of Elasticsearch sizing, considering variables such as data volume, the number of jobs, and job complexity. Testing these configurations is typically iterative and is best initiated in a lab environment using actual data over synthetic data. Single-metric jobs generally consume fewer resources compared to multivariate jobs, which demand more memory due to the number of variables involved. In production, dedicated machine learning nodes are recommended for optimal performance, with guidelines suggesting at least 4 cores and 64GB RAM per node. Machine learning jobs in Elasticsearch influence data nodes primarily through data retrieval and anomaly score re-normalization processes, with production environments benefiting from multiple dedicated ML nodes to enhance high availability and manage job distribution. Key configuration settings in the elasticsearch.yml file allow for the management of ML nodes and jobs, ensuring that the system can adapt to the demanding requirements of real-time and historical data analysis.

A new benchmarking tool for Logstash has been introduced, inspired by Rally's success with Elasticsearch, and it offers enhanced functionality by utilizing monitoring APIs built into Logstash since version 5.x. This tool requires only a JVM and can run benchmarks on predefined configurations and data sets, such as Apache, across various Logstash distribution versions or directly from GitHub sources. The output includes performance metrics like throughput and CPU usage, and results can be shipped directly to Elasticsearch for further analysis. Planned improvements include support for custom configurations and datasets, along with detailed documentation. In addition to the benchmarking tool, several fixes and performance improvements have been made in the Logstash core and plugins, including bug fixes in the Dead Letter Queue input, enhancements in metrics calculation, and optimizations in throughput when using in-memory queues.