January 2019 Summaries

The Query Suggestion API, now available in Elastic App Search, enhances search experiences by suggesting strong queries as users type, helping them find the right words and improving result relevance. This is particularly useful when users struggle to remember exact names, such as in the example of searching for Zion National Park. The API suggests possible matches like "zion" when a user inputs a vague query like "zeyeon," guiding them toward effective searches. Along with other tools like Synonyms, Relevance Tuning, and Curations, which refine search results by grouping similar terms, boosting relevance based on field values, and managing query results, this API is part of a comprehensive suite designed to optimize search functionality. These features are accessible through API and user-friendly dashboards, allowing for deep relevance tuning and enhanced search capabilities. Elastic offers a 14-day free trial to explore these features, preloaded with a National Parks data set, enabling users to experiment before integrating their data.

Application Performance Monitoring (APM) is a crucial component of observability, working alongside logs and infrastructure metrics to provide comprehensive insights into application performance. Logs record events and are human-readable, while metrics provide periodic summaries, both focusing on infrastructure rather than application specifics. APM bridges these gaps by focusing on the application layer, enabling developers to monitor end-user experiences and identify bottlenecks and errors. Elastic APM, integrated into the Elastic Stack, offers a curated user interface with visualizations and dashboards, allowing users to track transactions, traces, and exceptions. It provides detailed context about application performance, helping prioritize fixes and optimize performance. Elastic APM is open-source and integrates with Elasticsearch, allowing users to deploy it easily via Elastic Cloud or locally, and it supports various licensing levels for additional features like alerting and machine learning.

Elastic App Search has introduced role-based access control for its premium plans, allowing users to efficiently manage large teams by assigning one of six pre-defined roles, which include Owner, Admin, Dev, Editor, Analyst, and Finance. This feature enhances user management by enabling teams of up to 100 people to organize securely across multiple projects, with each role offering varying levels of access and control over account features such as billing, document curation, and analytics. Elastic App Search is designed to facilitate the integration of search functionalities into applications by providing robust tools for data indexing, relevance tuning, and analytics through an intuitive dashboard and APIs. Users can quickly set up teams by sending email invitations and managing access levels via the Users menu, with additional guidance available in the Role Based Access Control Guide. A 14-day free trial is available for those interested in exploring its capabilities without the need for a credit card.

The release of Elastic APM 6.6.0 introduces several enhancements aimed at improving application performance monitoring. Key features include the automatic collection and visualization of system and process-level CPU and memory metrics from various backend agents, such as Java, Node.js, Go, Python, and Ruby, now accessible through the new Metrics tab in the APM UI. Distributed tracing, previously in beta, is now generally available and allows users to trace requests across distributed service architectures, with support for JavaScript RUM and compliance with W3C Trace Context standards. The integration of APM with new logging and infrastructure apps enables real-time log streaming and infrastructure snapshots, facilitating faster debugging through data correlation. Additionally, the Java agent now supports wildcard tracing and easier agent attachment, simplifying the instrumentation of JVMs without altering server configurations. Elastic APM is also available on Elasticsearch Service, providing expanded deployment options with a 14-day trial and a free APM Server.

Elastic has announced the general availability of its Application Performance Monitoring (APM) Server integrated with the Elasticsearch Service, offering enhanced application performance insights and visibility within distributed workloads. This integration supports various programming languages, including Java, Go, Ruby, Python, and JavaScript, and builds upon existing features of the Elasticsearch Service, such as automated anomaly detection using machine learning and efficient troubleshooting with alerting capabilities. The service includes a cost-effective hot-warm architecture with built-in index curation and centralizes logs, metrics, and APM data. Current Elasticsearch Service users can add APM Servers to deployments free of charge, with a 512 MB APM Server instance included in all new 6.6+ deployments as part of the free allowance, allowing users to instrument applications and manage numerous events per second. New users can experience the service through a 14-day free trial.

Elastic Stack 6.6.0 introduces several enhancements that simplify cluster management and improve performance across various components, such as Elasticsearch, Kibana, and APM. Key features include the new Index Lifecycle Management, which automates the data lifecycle into hot, warm, cold, and delete phases, optimizing storage and reducing costs. Frozen indices are introduced to allow higher storage-to-memory ratios by trading search latency for storage efficiency, beneficial for long-term data archiving. The update also brings faster geoshape indexing using Bkd-backed data structures, enhancing storage and memory efficiency. Elasticsearch SQL gains support for date histograms, aiding in time-based analyses, while machine learning now offers annotations for better collaboration during system investigations. APM expands its capabilities with new agent metrics and improved integration with OpenTracing. Additionally, numerous improvements are made to Beats, Logstash, and Kibana, including Auditbeat's new system module for security data, Filebeat's NetFlow input, and enhanced Kibana connectivity and visualization features. Overall, Elastic Stack 6.6.0 offers a range of tools and improvements designed to optimize data management and system performance.

The 6.6.0 release of Logstash introduces the initial phase of native support for Java plugins, allowing developers to write input, filter, and output plugins using Java without relying on Ruby's ecosystem. This update aims to provide first-class support for pure Java plugins, enabling them to function seamlessly alongside Ruby plugins within the same Logstash pipeline and optimize performance by minimizing the interoperability cost between JRuby and Java. Although this feature is still experimental and does not yet offer full parity with Ruby plugins, the Java execution engine is being promoted to GA, and full feature parity is expected in a future release. Developers can experiment with Java plugin development, and feedback is encouraged through community channels.

File Data Visualizer, introduced in Elastic Stack 6.5, is a feature designed to simplify the import of delimited files, NDJSON, and semi-structured text into Elasticsearch, facilitating data exploration in Kibana and Machine Learning without deep technical knowledge. It utilizes the find_file_structure endpoint to analyze file content, suggest ingest pipelines, and create mappings for seamless data integration into Elasticsearch. Users can configure settings such as data format, delimiter, and field names through an intuitive interface, offering both simple and advanced import modes. Although currently experimental and not recommended for production processes, the tool allows users to import data into new indices and provides insights through Kibana for further analysis. The feature is particularly beneficial for users with limited technical backgrounds, as demonstrated by a marketing team member who successfully imported and visualized earthquake data. The tool also supports importing semi-structured log files, using grok patterns to identify fields, and offers the ability to edit field names and types for accuracy.

Greg Paskal leads the Quality Assurance Test Automation efforts at Ramsey Solutions, focusing on enhancing quality and reducing risk through innovative test automation. After attending the Google Test Automation Conference in 2016, he was inspired to improve the measurement and analysis of testing data. This led to the integration of the Elastic Stack and Kibana to create a "data lake" that captures and analyzes detailed test automation metrics. This approach enables the team to gain valuable insights into the software development lifecycle, going beyond simple pass/fail results to include test duration and other metrics. Paskal highlights the importance of effective data visualization and communication, drawing from the teachings of experts like Edward Tufte, to ensure the information is actionable and understandable for different audiences. The implementation of the Elastic Stack has significantly improved the team's ability to monitor application health and detect issues early, prompting interest and requests to share their methodologies within Ramsey Solutions.

The text provides an update on recent developments within the Logstash project, particularly emphasizing its support for JRuby 9.2 as part of efforts to ensure compatibility with Java 11. After overcoming previous issues with JRuby 9.1.13.0, the stable release of JRuby 9.2.5.0 has allowed progress to resume, aided by improvements in compatibility with newer Bundler versions. The Logstash CI now includes two manual Jenkins jobs that simplify release management tasks, such as drafting release notes and updating plugin versions. The text also details changes in various Logstash plugins, highlighting the removal of obsolete fields and options, and provides information on the latest commits and documentation updates across multiple Logstash repositories.

In the release of version 6.5 of the Elastic Stack, two new beta interfaces, the Infrastructure and Logs UIs, were introduced to enhance data interaction. The Logs UI provides a user-friendly way to access log data similar to the traditional tail -f command, allowing users to filter logs by criteria such as errors or specific labels, thus simplifying the process of dealing with large volumes of data. Configuration is straightforward, involving adjustments to the Kibana configuration file to integrate various data sources. On the other hand, the Infrastructure UI is designed to help operations teams manage infrastructure by visualizing key indicators like CPU, memory, and traffic for hosts, Kubernetes pods, and Docker containers. This UI supports grouping and filtering to allow users to focus on specific device subsets, with minimal setup required if using default configurations. Both UIs aim to streamline data management and visualization, and feedback is encouraged to refine these tools further.

Logstash is an open-source, server-side data processing pipeline designed to ingest, transform, and route data to multiple outputs. The blog post by Alex Marquardt illustrates how to use Logstash to process data from various stock markets and direct it to distinct outputs. By using a CSV file with stock market benchmarks as input, the process involves converting CSV rows to JSON documents, creating document copies with the clone filter plugin, and filtering each document to retain only the relevant fields for specific stock markets like "SMI" and "FTSE". Metadata is added to guide the documents to their respective Elasticsearch outputs, which are indexed separately. This example demonstrates Logstash's capability to handle complex data processing tasks without relying on the beta pipeline-to-pipeline communication feature. The post also provides a detailed Logstash pipeline example that showcases the steps necessary for setting up and executing the described data processing workflow.

Rollup data in Kibana allows users to efficiently manage storage by summarizing and compacting historical data into a single document using Elasticsearch's rollup APIs, which helps reduce storage costs as data volume increases. This process involves selecting specific fields for future analysis, creating a new rollup index, and maintaining the original data alongside or archiving it. Kibana version 6.5 introduces features for creating, managing, and visualizing rollup jobs, enabling users to start, stop, and delete these jobs, though changes to rollup job settings require creating a new job. Visualizations in Kibana can incorporate both rolled up and raw data, allowing for a comprehensive analysis using various chart types, with certain exceptions, such as Timelion and Visual Builder. Additionally, combination index patterns can be employed to visualize live and rolled up data together, enhancing the ability to analyze data while conserving storage resources.

Alex Francoeur's blog post celebrates his brother Mike's remarkable achievement of completing the Pacific Crest Trail, a 2,650-mile trek from Mexico to Canada, and details the use of the Elastic Stack to track and analyze his journey. Despite facing challenges such as injuries, harsh weather, and solitude, Mike persevered, completing the trail in 190 days and earning admiration from his family and the Elastic community. Francoeur used Elastic's machine learning to analyze data from Mike's hike, identifying anomalies in his daily mileage and overcoming challenges posed by limited GPS data and API inaccuracies. The blog also highlights Elastic's Canvas tool, which Francoeur employed to create compelling visualizations of Mike's journey, illustrating how the trail transformed him and marking a significant milestone coinciding with Elastic's public offering. The series underscores the supportive role of technology in adventure tracking and celebrates the broader community's engagement with Mike's story.

Tony Sleva recounts his personal experience of a significant earthquake in Alaska and the subsequent aftershocks, using it as a backdrop to introduce a basic tutorial on importing and visualizing earthquake data with the Elastic Stack. Despite being an entry-level guide, the blog demonstrates how even novices can use Elasticsearch and Kibana to analyze data, in this case, earthquake data from the USGS, by importing it as a CSV file. Sleva describes the process of setting up a free trial with Elasticsearch Service, importing the data into Kibana, and exploring pre-made visualizations to better understand the frequency and magnitude of aftershocks, ultimately finding comfort in the data trends showing a decline in aftershock activity. Through this experience, he emphasizes the accessibility and utility of the Elastic Stack for data analysis, even for those without a technical background.

AmericanAncestors.org, operated by the New England Historical Genealogical Society, enhanced its user search experience by transitioning from SQL to Elasticsearch, significantly improving search flexibility and speed for its 250,000 members who access over 1.4 billion names and historical documents. Initially, the system ran on outdated Elasticsearch 1.3 across five physical servers set up by contractors, which became inefficient and unstable. Under the leadership of Don LeClair, the organization purchased a Platinum subscription to the Elasticsearch Service, optimizing their clusters, reducing system administration time, and implementing security measures. This new approach allowed for a more dynamic search system that accommodated suggestions, although it initially increased server workload, especially after vulnerability patches. Collaboration with the Elastic support team led to further optimizations, including identifying problematic queries and improving data management, resulting in a 60% reduction in disk usage and a 50% decrease in CPU usage, making searches 2-3 times faster. These improvements enabled AmericanAncestors.org to handle a 400% traffic surge during a promotional event with ease, while also reducing costs by 25% compared to their previous setup.