May 2018 Summaries

Monitoring applications running in Docker containers and Kubernetes requires a dynamic approach, as the system must adapt automatically to scaling changes without burdening administrators. Docker containers offer a lightweight, isolated environment for software with all necessary components, while Kubernetes facilitates the automated deployment and management of these containerized applications. Traditional monitoring definitions are expanded in this context to include collecting, parsing, storing, visualizing, and acting on logs and metrics from applications and their environments. The Elastic Stack, comprising Elasticsearch, Kibana, Beats, and Logstash, simplifies these tasks by providing modules for common log formats and leveraging container metadata for autodiscovery. This enables quick setup of visualizations and automated alerts, illustrated through a practical example in the IBM Cloud Container Service. Future discussions aim to delve deeper into the specifics of Elastic modules and configuration techniques for efficient monitoring.

Elastic Cloud Enterprise (ECE) is a robust orchestration tool that helps enterprises centrally manage their Elastic Stack deployments, facilitating internal teams to quickly derive insights without the hassle of handling Elastic Stack operations. The platform offers a powerful API that allows users to monitor and understand resource allocation and usage across different teams, which is crucial as more teams onboard to a centralized ECE environment. This commentary provides a guide on leveraging ECE's API to gather usage statistics, such as memory and disk usage, of Elasticsearch clusters, enabling administrators to track changes over time. By utilizing X-Pack’s Alerting Watcher capability, users can set up periodic API calls to capture data, which is then indexed and visualized in Kibana for comprehensive insights. Though the post discusses deprecated features, it showcases methods to create visualizations of memory and disk usage, with intentions to expand metering capabilities to include network bandwidth and usage metrics in the future.

In the blog post "Seven Tips for Better Elasticsearch Benchmarks," Daniel Mitterdorfer outlines strategies to enhance the accuracy and reliability of benchmarking Elasticsearch, emphasizing that these principles are applicable to various systems. The author highlights the importance of system setup, advocating for benchmarks that closely mirror production environments to ensure valid results. A thorough warm-up phase is necessary to stabilize performance metrics, while workload modeling should reflect real-world usage patterns to yield realistic outcomes. The article acknowledges the potential for bugs in benchmarking software and stresses the need to identify and mitigate accidental bottlenecks that can skew results. A structured process involving controlled variables and environment resets is recommended to maintain consistency, and statistical analysis is crucial to account for run-to-run variations. These insights aim to guide practitioners in obtaining meaningful performance metrics that inform decision-making, avoiding common pitfalls in the benchmarking process.

In the May 28, 2018, edition of "Keeping up with Kibana," several updates and enhancements to the Kibana project are highlighted, including the introduction of a new one-click Sample Data feature that allows users to experiment with Kibana using a pre-packaged Flight data set and dashboard. The Canvas technology is undergoing a transformation with a shift to Elastic's in-house UI framework (EUI) and improvements in layout engine capabilities, including 3D transforms and keyboard commands. Security and management updates include progress on role-based access control (RBAC) and the introduction of the SavedObjectRepository to enhance security and functionality. Other notable enhancements include compatibility with React 16.3, improvements to the EuiSearchBar, and operational updates such as the option to run Kibana from build for continuous integration. The article concludes with various bug fixes and updates across different sections of the Kibana platform, ensuring enhanced usability and performance.

The recent updates in Elasticsearch and Apache Lucene highlight several key changes and improvements aimed at enhancing functionality and performance. Elasticsearch 7.0 will have HTTP pipelining permanently enabled to prevent undefined behaviors, while context is being made mandatory in the context suggester for better efficiency. Keyword splitting on whitespace at query time is now an optional feature to cater to user preferences, and plugin signature verification has been introduced to ensure the authenticity of plugins. Cross-cluster replication benchmarking has progressed significantly, with the team successfully transferring 30GB of data across regions in Google Cloud. Improved authentication handling has been implemented to prevent redundant requests, and security fixes, dynamic mapping updates, and enhancements in query performance are also part of the changes. Additionally, the Lucene impacts for synonym and phrase queries have been optimized, promising faster search results. Discussions are ongoing about handling large document numbers in indices and improving internal functionalities such as the IndexWriter and token filters.

The tutorial provides a comprehensive guide to setting up an Elasticsearch environment on Elastic Cloud, using a sample Apache log dataset to illustrate the process. It begins with instructions on how to create an Elastic Cloud account and deploy a hosted Elasticsearch cluster, followed by steps to secure the cluster using Kibana and the X-Pack security module. Users are guided through the data import process using Filebeat and Apache logs, with detailed instructions for verifying data ingestion and cluster health. The tutorial concludes with visualizing the imported data in Kibana and suggests further resources for exploring additional datasets or integrating personal data using tools like Logstash or Beats.

Kibana allows users to customize basemaps for region and coordinate maps beyond the default Elastic Maps Service, which uses data from Natural Earth and OpenStreetMap. Users can configure alternative basemaps through the kibana.yml file by using a custom tile-server or a Web Map Service (WMS) server. Various third-party services like MapTiler, Mapbox, ArcGIS Online, and Wikimedia provide APIs for tile maps that can be integrated with Kibana, each requiring proper attribution to data sources like OpenStreetMap. Users can also opt to self-host basemaps if Elastic Maps Service is unavailable or if they operate in isolated environments. The process involves configuring parameters like map.tilemap.url, map.tilemap.options.attributions, and map.tilemap.options.maxZoom in the kibana.yml file, ensuring that the maps meet specific requirements such as raster format and Web Mercator coordinate system. These configurations enable flexibility and creativity in visualizing geospatial data within Kibana dashboards.

In this document, the challenges and solutions for searching Chinese, Japanese, and Korean texts using Elasticsearch 6.2 are explored, focusing on the need for custom language analyzers. It highlights that the default analyzer, known as the standard analyzer, is not optimal for these languages due to their agglutinative nature and the use of complex scripts, which results in poor tokenization. The text provides examples of how the standard analyzer performs inadequately by combining nouns with postpositions in Korean and Japanese or breaking down Chinese characters into individual tokens. To address these issues, the document introduces language-specific analyzers such as kuromoji for Japanese, smartcn for Chinese, and openkoreantext-analyzer for Korean, which offer improved tokenization by correctly handling postpositions and maintaining the integrity of words and characters. These custom plugins need to be installed on every node in the Elasticsearch cluster to enhance search capabilities, and their effectiveness is demonstrated through examples with improved tokenization results.

Logstash is a versatile, plugin-based data collection and processing engine that enables users to efficiently parse, process, and forward data into Elasticsearch using customizable pipelines and configuration files. This blog post provides a detailed introduction to Logstash, focusing on its ability to handle various log formats with minimal configuration through plugins like Filebeat, and showcases the development of a configuration for parsing Squid cache access logs. Using both the dissect and grok filters, Logstash can parse logs into distinct fields, with grok offering the flexibility of regular expression matching for more complex log formats. The post also highlights the importance of type conversion and the use of the date filter to ensure data is appropriately formatted before being sent to Elasticsearch, where specific mappings can be applied to fields to optimize data retrieval and aggregation. The article underscores Logstash's role as a powerful tool in data processing pipelines, encouraging readers to explore further resources and examples to fully leverage its capabilities.

In the weekly update titled "Keeping up with Kibana," the development team shares key advancements in various projects within the Kibana ecosystem. The Platform team is working on integrating first-class TypeScript support as part of their new platform roadmap, which will also support new plugin development like the Infra app. Progress is being made on localization, security enhancements with role-based access control (RBAC), and the Spaces implementation, as well as a focus on improving accessibility. The team is preparing for the upcoming 6.3 release, with quality assurance actively testing the new features. Additionally, updates include improvements in schema handling, visualization capabilities, and management features, alongside functional test runner changes and addressing issues such as timing and control order modifications.

In the second part of a series on enriching postal addresses using the Elastic Stack, David Pilato explores how to utilize the BANO dataset for efficient address lookup and enrichment. The blog demonstrates querying the BANO index with Logstash to search for French postal addresses using both text-based and geo-point searches, optimizing search efficiency by leveraging department numbers. Pilato introduces a Logstash enrichment pipeline that enables hot-reloading to improve developer efficiency and outlines how to use the elasticsearch-filter-plugin to integrate Elasticsearch queries into Logstash for enriching datasets with location-based data. He also addresses the complexity of French postal codes, particularly those starting with '97', and provides solutions for handling them efficiently within the pipeline. Finally, he suggests practical applications of this technique for enriching existing datasets, setting the stage for further exploration in the series.

In this third installment of a series on enriching postal addresses using the Elastic Stack, David Pilato demonstrates how to enhance an existing dataset by integrating the BANO dataset with Logstash. The process begins by reading a CSV file with Filebeat instead of using the http-input plugin, configuring a beat-input plugin to handle file input. A CSV filter is then applied to parse the data, which includes geolocation points, and enrich it by sorting based on geographical distance. The enrichment process achieves a rate of approximately 140 documents per second, with an average event latency of 20-40 ms. Despite some slowdown due to Elasticsearch lookups, the method remains efficient for ETL operations compared to using Elasticsearch as an ingest pipeline. Additionally, Pilato suggests alternatives like reading data from SQL databases using a jdbc-input plugin or connecting to existing Elasticsearch data for further enrichment. The series concludes with the prospect of indexing other open data sources to cover regions beyond France.

The blog post outlines the process of using the Elastic Stack, specifically Logstash, to enrich postal address data in France by leveraging the Base d'Adresses Nationale (BANO) dataset. The author describes the challenges of incomplete and poorly formatted address data, which complicates tasks like mapping customer locations and retrieving addresses from geographic coordinates. To address these issues, the post discusses the drawbacks of using external services like Google Maps API, such as latency and data privacy concerns, and instead proposes a local solution using BANO, an open dataset under the OpenStreetMap umbrella. The post details steps to download BANO CSV files for French departments, parse and load the data into Elasticsearch, and set up index templates to handle address data efficiently. Additionally, it explains the use of custom analyzers for address normalization and the creation of Logstash pipelines to transform and store the data. The result is a searchable, enriched dataset that can be visualized using Kibana, providing insights into address distribution and enabling tasks such as address correction and transformation.

In a recent update on Elasticsearch and Apache Lucene, several enhancements and changes were highlighted, including a reduction in the default number of shards per index to address oversharding, the addition of support for the Search Template API in the high-level REST client, and the removal of custom key and trust managers to comply with FIPS standards. The update also mentions improvements such as faster histograms with non-fixed time zones to benefit Kibana users, optimization of top-k queries by storing static scoring signals as term frequencies, and progress on cross-cluster replication with added security permissions. Additionally, a new NIO-based transport layer has been introduced, and changes have been made to improve caching efficiency and forbid expensive query parts in ranking evaluation. Apache Lucene's recent release, version 7.3.1, addressed issues like removing the 2 billion document limit per index, better handling of pending deletes, and allowing doc-value updates to non-existing fields. The update reflects ongoing efforts to enhance efficiency, security, and functionality in both Elasticsearch and Apache Lucene.

The blog post by Kuniyasu Sen provides a detailed guide on setting up TLS for Elasticsearch, Kibana, Logstash, and Filebeat on a two-node cluster using an offline installation on CentOS 7. It addresses how to perform this setup using RPM packages for environments where direct internet access is unavailable, emphasizing the importance of securing communication within the Elastic Stack. The process involves installing Elasticsearch and its X-Pack plugin, creating SSL certificates, enabling TLS, and configuring each component for secure data transport. The guide is aimed at demonstrating a sandbox setup and advises further adjustments for production environments, such as using a minimum of three nodes for high availability and commercially signed certificates. Despite the guide being based on older Elastic Stack versions 6.2.3, it serves as a foundational reference for setting up secure, offline installations, with a note that it uses deprecated features and links to more current documentation for Elastic Stack 6.8/7.1 or later.

At Elastic{ON} 2018 in San Francisco, the Elastic Training team organized a drawing for an Online Annual Training Subscription, which was open to attendees who had their badges scanned at the event. The process involved using Elasticsearch to handle the data, ensuring fairness and privacy throughout. They began by removing duplicate entries and filtering out Elastic employees, then redacted personally identifiable information before indexing attendee emails. By using Elasticsearch's _bulk API and a specialized ingest pipeline, they assigned document IDs, extracted email domains, and routed entries based on eligibility, ensuring all potential personal data was redacted. The eligible entries were then indexed into a single-sharded database to prevent data inconsistency, and a random score query with a seed value ensured a fair selection process. Ultimately, Stephen Steck was announced as the winner, receiving the training subscription prize. The entire process, including technical details and sample data, was shared on a Github Gist for public exploration.

Dr. Suraj Pandey's article discusses using the Elastic Stack, specifically Kibana, to visualize the 2018-19 Australian budget data, which can otherwise be daunting to interpret in its raw form through traditional spreadsheet applications. Kibana offers an interactive method to explore multi-dimensional budget data, allowing users to drill down into specific areas such as departmental expenses, making the data more accessible and engaging. The article guides readers through the technical process of setting up visualizations with Kibana, using Logstash to import and transform CSV files into a format suitable for analysis. The approach simplifies data sharing and enhances the understanding of complex budget allocations by presenting them in an easily digestible visual format, demonstrating the power of interactive data visualization tools in making large datasets more user-friendly and insightful.

The Central Securities Depositories Regulation (CSDR) was introduced to enhance the stability and resilience of the European financial system by harmonizing the operations of Central Securities Depositories (CSDs) and implementing a standardised penalty regime for settlement fails. This regulation, which emerged in response to the 2007-2008 financial crisis, requires CSDs to monitor and report failed transactions, imposing cash penalties and a mandatory buy-in process to deter settlement failures. As part of compliance with CSDR, financial institutions like Goldman Sachs have utilized the Elastic Stack technology to create systems that track trade flows, providing real-time visibility, advanced search capabilities, and comprehensive reporting, thereby ensuring timely settlements and adherence to the regulation. The Elastic Stack's flexibility, scalability, and ease of integration make it an ideal solution for organizations aiming to meet the CSDR's stringent requirements, fostering more efficient trade tracking and system health monitoring.

NetApp, a leading vendor in all-flash array and integrated systems, enhanced its OnCommand Insight (OCI) monitoring tool by embedding Elasticsearch, significantly improving its functionality and customer value. Originally using MySQL, Lucene, and Cassandra for data management, NetApp transitioned to Elasticsearch to better handle data visualization and trend forecasting, particularly in hybrid cloud environments. This shift allowed OCI to provide advanced features like multiple timeline visualizations and hypervisor averages, which were previously unavailable. As part of their ongoing evolution, NetApp continues to integrate components of the Elastic Stack, such as Logstash and Beats, into OCI. This integration has been well-received, demonstrating the benefits of embedding Elasticsearch and machine learning to optimize operations and resource management, while also highlighting the need for expandable plugins to meet evolving data challenges.

Adaptive Replica Selection (ARS) is a feature in Elasticsearch designed to improve response latency by intelligently routing search requests to the most suitable shard copies, especially when some nodes experience performance degradation. Traditional round-robin routing can lead to inconsistent response times, as degraded nodes can significantly slow down search requests. ARS, based on a formula adapted from an academic paper originally written for Cassandra, evaluates various metrics like outstanding search requests and response times to rank shard copies, directing requests to those expected to perform best. This method ensures that requests are not consistently sent to overloaded nodes, thereby reducing tail latency and enhancing overall system performance. Benchmarks demonstrate that ARS improves both throughput and latency in scenarios with and without node stress, showing significant gains in handling loaded nodes by routing requests away from them. While ARS is available from Elasticsearch 6.1, it is turned on by default only in version 7.0 and later, allowing users to dynamically enable it in earlier versions to benefit from more efficient search request handling.

RS2 leveraged the Elastic Stack to enhance their security operations by reducing manual efforts in incident management and meeting compliance requirements. Initially, RS2 explored using the Elastic Stack, starting with Elasticsearch and Kibana on a virtual machine, to create a proof of concept (POC) demonstrating its effectiveness in handling data from network devices and third-party integrations. The POC, which involved parsing and enriching logs, creating custom dashboards, and implementing a Slack-based alerting framework, proved successful and led to the adoption of Elastic Cloud for scalability and security. The deployment allowed RS2 to manage data efficiently and seamlessly integrate additional data sources while continuously upgrading and expanding their usage of Elastic Stack to improve security analytics and operations.

The process of visualizing France's salary data using region maps in Kibana involves importing a Kaggle dataset on employment, salaries, and population, which is sourced from the French National Institute of Statistics and Economic Studies (INSEE). The data requires preparation by joining tables from different CSV files using SQLite to normalize INSEE codes before importing into Elasticsearch. Afterward, Logstash is configured to ingest this data into Elasticsearch, employing CSV filters to convert and format the data appropriately. In Kibana, a new index pattern is created to visualize salary data by French departments, allowing users to map average mean net salaries across regions and explore gender pay gaps through scripted fields. The visualization process involves customizing region maps to display data layers such as department name and salary ratios, providing insights into salary distributions and disparities across different demographics in France.

Jeff Whelpley describes how he rapidly integrated Elasticsearch into his company's spending tracker app, Swish, to enhance the categorization of bank transaction descriptors by associating them with company names. Faced with the challenge of inconsistent transaction descriptor formats, he leveraged Elastic's Elasticsearch Service, which facilitated quick deployment without the need for extensive management of infrastructure. He created an index for company data and used a simple dump-and-load approach to transfer data from MongoDB to Elasticsearch. By implementing prefix searches to match descriptors with company names, the team significantly improved their hit rate from 5% to 70%, enhancing user experience by providing clearer transaction information. Despite the complexities involved in setting up Elasticsearch, the swift and successful implementation underscored its value in improving app functionality and the user experience, demonstrating the potential for further skill development with the tool.