March 2017 Summaries

Aaron Mildenstein's article provides a detailed guide on using Logstash to create an Elasticsearch mapping template, which optimizes data storage and query efficiency by explicitly defining how Elasticsearch should interpret field data. He explains the advantages of replacing Elasticsearch's default "schema-less" indexing with customized mappings, particularly when managing large datasets, as it can reduce storage and memory requirements. By using a practical example involving Apache access log data, the article demonstrates how to configure Logstash to send data to Elasticsearch, create and edit mappings, and convert these mappings into a template. The guide also touches on specific field types, such as transforming IP fields into type "ip" and geo-location fields into "geo_point," to enhance the precision and functionality of data queries. Mildenstein concludes by providing instructions to upload and test the template, ensuring that the user's customized mappings are correctly applied in Elasticsearch.

Elastic Cloud offers a hosted Elasticsearch service that allows users to deploy the Elastic Stack without the need for provisioning hardware, providing on-demand Elasticsearch clusters with features like dynamic scaling and the latest Elastic Stack and X-Pack versions. Initially hosted on Amazon, Elastic Cloud can now be billed through either Elastic or AWS, thanks to its listing in the AWS Marketplace. This integration simplifies the payment process by allowing users to pay with existing AWS account information, consolidating all metered usage into a single bill. AWS's introduction of SaaS subscriptions has enabled users to discover, subscribe to, and consume various SaaS and API products, including Elastic Cloud, directly from the AWS console. The service offers eight deployment variants based on size and region, and provides full X-Pack capabilities, including monitoring, alerting, and security. Elastic Cloud is available for a 14-day free trial, encouraging feedback on the user experience.

Elasticsearch 5.3.0, based on Lucene 6.4.1, introduces several innovative features and improvements, including a fix for a security bug in X-Pack 5.2.x related to Field Level Security. The release enhances search capabilities with Cross-Cluster Search, allowing any node to act as a search client with dynamic remote cluster configurations and lightweight connections, addressing limitations of the Tribe Node. The new unified highlighter offers improved snippet highlighting by fixing bugs and supporting different analysis methods. Field collapsing is now more efficient, easy to use, and supports pagination, while additional features include an adjacency aggregation for filter matches and a keystore for secure settings, signifying a step towards better security management. Users are encouraged to explore these updates and provide feedback.

At Elastic{ON} 2017, an innovative keynote performance featured a ballerina outfitted with sensors to capture biometric and motion data, which was visualized in real-time using the Elastic Stack. The project, known as the Ballerina Project, aimed to demonstrate the creative and unexpected applications of Elastic's technology by visualizing heart rate, acceleration, location, and sound data during a ballet performance. Despite challenges like embedding technology into the ballerina's costume and ensuring the visualizations were understandable to the audience, the project highlighted how the Elastic Stack can transform data into compelling stories. The successful execution of this performance underscored Elastic's commitment to pushing technological boundaries and inspiring curiosity and innovation among its audience.

At the Elastic{ON} 2017 conference, attendees were captivated by an expansive timeline wall that documented nine years of Elastic's history, from its nascent stages to the latest software release in January 2017. The timeline, which spanned the length of a blue whale, was meticulously crafted with over 700 guidelines and divided into sections highlighting software downloads, product releases, user group formations, and significant events within the company. Despite initial uncertainties about its impact, the timeline became a focal point for discussion and reflection, encouraging attendees to engage with Elastic's evolution by marking their personal milestones with neon stickers. The timeline's design drew inspiration from subway maps, illustrating the progression from sporadic releases to a more synchronized product integration, notably with the release of Elasticsearch 2.0 and subsequent updates. The installation served as both a visual narrative and a conversation starter, inviting attendees to explore Elastic's journey through an interactive and thought-provoking display.

In March 2017, Elastic Cloud faced a series of memory and connectivity issues affecting a small percentage of its Elasticsearch clusters, particularly those with low-memory nodes. These issues were due to a combination of small problems that collectively led to high memory pressure, increased garbage collection load, and frequent out-of-memory errors, particularly under certain conditions involving specific kernel and Docker version combinations. The problems were exacerbated by long-running containers and the interaction between the JVM and Linux memory management. Elastic Cloud identified and addressed these issues through various fixes, including upgrading kernel and Docker versions, adjusting JVM and Elasticsearch settings, and expanding their test suites to better simulate real-world conditions. These efforts were part of a broader strategy to enhance system reliability and support affected customers, especially those with smaller clusters and trial accounts.