June 2017 Summaries

Rightmove, the UK's leading property portal, transitioned to a microservices architecture to manage the increased demand and complexity of its operations, resulting in a significant increase in log data. To address the challenges of managing this data, Rightmove adopted Elasticsearch, which offered superior scalability, performance, and ease of use compared to its previous search engine. The implementation of the Elastic Stack, including Logstash and Kibana, has streamlined the process of collecting, monitoring, and analyzing application metrics and logs, greatly improving productivity and system reliability. This transition has facilitated faster problem-solving, easier tracking of system behavior, and better prediction of scaling needs, as well as empowered support teams to handle anomalies more effectively. Through internal training and advocacy, Rightmove has successfully integrated the Elastic Stack into its development workflow, resulting in a more efficient and responsive system that meets both internal and user needs.

Alex Francoeur and Chris Cowan present a video tutorial showcasing the versatility and power of Kibana's new Time Series Visual Builder, an advanced user interface for visualizing time series data. The tutorial guides users through various features, including the ability to quickly change visualization types using the same data set, leverage overall functions for aggregating data over a total time interval, and utilize the data formatter to switch between data types seamlessly. Additionally, it demonstrates how to apply conditional styling to metric visualizations based on the latest values of time series data. To fully engage with the tutorial, users are encouraged to download Metricbeat for ingesting system or service metrics and ensure they have the latest version of Kibana installed. The video is part of a series, with more tutorials anticipated to explore the capabilities of the time series visual builder further.

Elastic's acquisition of Opbeat marks a strategic move to enhance its capabilities in application performance monitoring (APM) by integrating Opbeat's developer-focused tools with the Elastic Stack. The partnership aims to leverage Opbeat's expertise in monitoring JavaScript ecosystems, such as Node.js, AngularJS, and React, to provide developers with comprehensive insights into application performance and code issues. This integration will enable users to analyze infrastructure logs and performance data on a single platform, enhancing the Elastic Stack with new features like anomaly detection and machine learning. Both companies share a cultural alignment and a commitment to improving DevOps practices by offering a unified APM solution that supports modern development needs. Current Opbeat services will remain operational until the Elastic APM product is launched, ensuring a smooth transition for existing customers.

Elastic Cloud Enterprise (ECE) enhances the management and security of Elasticsearch clusters by supporting automated snapshots to user-defined repositories, such as AWS S3-compatible storage. This feature allows for incremental, point-in-time snapshots that are taken every 30 minutes and before any cluster topology changes, ensuring that data can be quickly restored in the event of unintended or destructive changes. ECE stores the last 100 snapshots per cluster, providing roughly two days' worth of backup retention. The automated snapshot process is efficient and minimally impacts cluster performance, making it a practical solution for managing multiple clusters. While currently supporting S3-compatible storage, ECE plans to expand its support to other storage options like Google Cloud Storage and Microsoft Azure Storage.

Elasticsearch 5.4.2, a bug fix release based on Lucene 6.5.1, is now available and deployable on Elastic Cloud. Key improvements include resolving issues where installations using systemd with X-Pack could hang during shutdown, preventing out-of-memory errors in aggregations by better managing memory allocation, and optimizing memory usage in terms aggregations that sort on sub-aggregations. Additionally, geo-point fields from indices created in version 2.x now function correctly with the field stats API. Users are encouraged to download the update, provide feedback via Twitter or forums, and report any issues on GitHub.

Filebeat modules, introduced by Elastic, simplify the ingestion and analysis of common log formats by providing preconfigured ingest pipelines and Kibana dashboards, which can be customized. This blog post delves into the on-disk storage requirements when indexing data using Filebeat modules, particularly focusing on the Apache2 module. By default, Filebeat modules offer efficient storage, saving around 20% disk space over standard Elasticsearch settings, with potential savings up to 50% through more aggressive tuning. The modules streamline the collection, parsing, and visualization of logs, and come with optimized mappings that minimize space usage without losing functionality. Various techniques to optimize storage include reducing unnecessary fields, enabling more aggressive data compression, and disabling certain flexible search features, each with its own trade-offs. The post highlights that while default settings already improve storage efficiency, further optimizations can be tailored to specific use cases for even greater space savings.

Elastic Cloud Enterprise (ECE) extends Elastic's cloud-based Elasticsearch and Kibana management to private datacenters, offering the same reliable platform used in Elastic's cloud service. This platform benefits from years of experience managing clusters at scale and includes tools like the Activity Feed and Cluster Activity view to streamline operations. The Activity Feed acts as an event log, displaying recent actions and pending tasks for clusters, which aids site reliability engineers (SREs) in managing vast amounts of data. The Cluster Activity view provides detailed historical insights into cluster changes, including unsuccessful attempts, enabling engineers to understand the current state and past configurations of clusters more effectively. Elastic continues to enhance its user interface, exploring more visual and digestible ways to present cluster histories, thereby improving the efficiency of the SREs' workflow.

Elastic Cloud Enterprise (ECE) 1.0.1, a minimal maintenance release, was announced to address security vulnerabilities in Kibana and Elasticsearch 5.4.0, ensuring that new ECE installations default to version 5.4.1 of these components. Users with version 1.0.0 do not need to upgrade their ECE installation but are advised to add the Elastic Stack pack for version 5.4.1 and upgrade their clusters accordingly, especially the admin-console-elasticsearch and logging-and-metrics clusters, as well as any existing 5.4.0 clusters. Although there is no supported method for deleting Elastic Stack packs, it is recommended not to create new 5.4.0 clusters. The installation process for version 1.0.1 remains the same as version 1.0.0, with further details available in the ECE installation documentation.

Outbrain, a leading content discovery platform, transformed its content recommendation system into a scalable search problem using Elasticsearch to handle its vast and complex requirements. The system indexes articles as separate Elasticsearch documents with semantic features to determine relevance, and user interests are translated into Elasticsearch queries, allowing for personalized content recommendations. Market rules, such as geographic targeting, are implemented as filters in Elasticsearch to ensure compliance. To incorporate advanced machine learning models, Outbrain developed custom Elasticsearch plugins, optimizing for high throughput and low latency by separating indexing from querying, utilizing a read-only index, and employing techniques like force merging and efficient data node management. These efforts allowed Outbrain to serve 800,000 requests per minute at under 100 milliseconds latency, enhancing the personalization and relevance of its recommendations while maintaining scalability and efficiency.

Elasticsearch 6.0.0-alpha2 has been released as the second pre-release version in preparation for the upcoming 6.0.0 release, built on Lucene 7-SNAPSHOT. This alpha version is intended solely for testing purposes, and users are encouraged to provide feedback on any issues encountered. Notable features in this release include the introduction of the significant_text aggregation for more efficient text field analysis, the new ip_range field type for indexing IPv6 address ranges, and the icu_collation_keyword field type for language-specific sorting. Additionally, this release continues efforts to remove support for multiple mapping types and improve fast replica recovery through sequence numbers. Users are advised against deploying this version in production environments, as indices created will not be compatible with the final 6.0.0 release, and upgrading from 6.0.0-alpha2 to other versions is unsupported.

Approaches for de-duplicating data in Elasticsearch using Logstash are crucial to ensure data integrity and accurate analytics, especially when dealing with metrics where duplicates can lead to incorrect aggregations and alerts. Elasticsearch's indexing process allows users to either provide a unique document ID or let Elasticsearch generate one, impacting how duplicates are handled. By using the Logstash fingerprint filter, users can create a unique fingerprint for events, which can be used as the document ID to prevent duplicates. This process involves generating consistent hashes from specific fields, such as the message field, using algorithms like MURMUR3 or cryptographic hash functions. For scenarios involving accidental duplicates, especially in persistent queue systems, generating UUIDs at the producer level ensures unique identifiers for each event, preventing duplication during reprocessing. This method emphasizes the importance of handling duplicates efficiently in data pipelines to maintain accurate and reliable data systems.

Elasticsearch has released versions 5.4.1 and 5.3.3, which include critical bug fixes and security patches, particularly for users of X-Pack Security. Version 5.4.1, based on Lucene 6.5.1, is the latest stable release and is available for deployment on Elastic Cloud. It addresses security issues such as a document level security bug with index aliases and a privilege escalation flaw in the run_as functionality. Users of X-Pack Security from versions 5.0.0 to 5.4.0 are advised to upgrade to these latest releases to mitigate these vulnerabilities. Additional improvements include addressing bugs related to single-shard scrolling, LDAP user caching, Netty resource allocation, and transport client sniffing protocol. The release also highlights enhancements in index setting validations and resolves issues with HDFS repository plugins and snapshot retrieval speeds. Users are encouraged to download the new versions, provide feedback, and report any issues on the GitHub page.

Kibana's new time series visual builder is an innovative UI designed to enhance the visualization and analysis of time series data by leveraging Elasticsearch's pipeline aggregations. This tool allows users to create and customize dashboards with features such as trend analysis of multiple index patterns, complex calculations using metric and pipeline aggregations, and conditional coloring based on user-defined thresholds. Users can also compare data across different timeframes, switch between various visualization types, and add annotations to trend charts. The visual builder is part of a series of blog posts and video tutorials aimed at helping users get started, with feedback encouraged to improve its experimental features.