December 2016 Summaries

In a blog post reflecting on the holiday spirit at Elastic, Kristina Frost highlights various charitable initiatives undertaken by the company's employees across the globe. The post emphasizes the importance of giving and caring in challenging times, with employees participating in efforts such as Movember for men's health, supporting youth in Nicaragua, and contributing to local food banks and toy drives. Notable projects include fundraising for a water distribution system in Nicaragua, holiday gift collections with the Salvation Army, and participating in the #Lakaan campaign to raise awareness for pneumonia. Elastic as a company also contributed $40,000 to organizations like Doctors Without Borders and Feeding America, underscoring its commitment to making a positive impact on both local and international communities. The narrative celebrates the collective effort to support at-risk populations and encourages continued generosity and community engagement.

In December 2016, the Elastic Stack introduced a new visualization feature for Kibana called Tag Cloud, designed to enhance dashboards, reports, and presentations by visually representing data through collections of words or phrases sized according to a chosen metric. This feature allows users to map metric values to font sizes using linear, log, or square root scaling, offering flexibility in how data is presented. While tag clouds can be used for various applications, such as analyzing web traffic or visualizing themes like those in political speeches or fictional universes, users must consider space limitations and adjust settings to avoid overlapping or omitted tags. The Tag Cloud integrates seamlessly with Kibana's other features and can be added to dashboards for interactive data exploration, allowing users to filter and analyze data in a more engaging manner. Elastic Stack encourages feedback for further enhancements, signaling ongoing development for future Kibana visualizations in the 5.x releases.

Elasticsearch introduced the Profile API in version 2.2 to provide detailed timing information for search queries, allowing users to identify slow components without relying on intuition or experience. While the API is verbose and generates extensive outputs, it enables users to gather comprehensive diagnostics by collecting timings and method invocation counts for query components across shards and indices. The introduction of the Search Profiler, originally an internal tool, aimed to visualize and interact with these outputs, evolving from a hierarchical tree structure to a more user-friendly hierarchical list-view with percentage bars to effectively identify query hotspots. Since version 5.1, the Search Profiler is included in X-Pack Kibana, allowing users to input search queries and generate visualizations, with interactive features such as collapsing and expanding query timings. It is freely available with any X-Pack license, reflecting the belief that diagnostic tools should be accessible to all users.

The blog post discusses the process of monitoring a Kafka cluster using the Elastic Stack, specifically focusing on utilizing Filebeat to collect and parse Kafka logs, which are then indexed into Elasticsearch for visualization in Kibana. The setup includes a Kafka cluster of three nodes, where Filebeat is configured to collect logs from various Kafka sources, including garbage collection stats, and process them using Elasticsearch Ingest Node pipelines. These pipelines employ grok patterns and scripts to parse and convert log data, extracting valuable information such as Java exception details and memory usage metrics. The data is then visualized using Kibana dashboards, which provide insights into log levels, stack traces, and garbage collection metrics, allowing users to identify potential issues and monitor system performance. The post highlights the flexibility of the Elastic Stack in accommodating custom configurations and encourages further exploration of dashboard functionalities to enhance monitoring capabilities.

Elastic Cloud is a hosted service by Elastic offering the latest features and improvements of Elasticsearch and Kibana, along with X-Pack features like security and monitoring. It simplifies operations and reduces costs while being supported by Elastic's team. Migration to Elastic Cloud can be done through several approaches, such as loading data from external sources, reindexing from remote clusters, or restoring snapshots from storage like S3. Reindexing allows data to take advantage of new data structures and performance improvements, whereas snapshot restoration is less costly but doesn't benefit from these enhancements. The article also provides detailed steps for both reindexing and snapshot restoration, including configuration requirements and considerations for different environments, ensuring a seamless transition to Elastic Cloud. A free 14-day trial and an easy-to-use pricing calculator are available for those interested in exploring Elastic Cloud.

This blog series provides a comprehensive guide on integrating the Elastic Stack with ArcSight SIEM to enhance security analytics capabilities, detailing the process of sending Common Event Format data from ArcSight to Elasticsearch using Logstash and Kibana, and leveraging X-Pack for added functionalities. The series outlines steps for setting up the Elastic Stack, including the installation of Elasticsearch, Kibana, and Logstash with X-Pack, and configuring ArcSight Smart Connectors to send data to Logstash. It also emphasizes the importance of exploring security data through Kibana dashboards and visualizations, and discusses the role of X-Pack alerting in monitoring security events. Further, the series plans to extend into scaling the architecture, setting up alerts for common security threats, and employing machine learning for anomaly detection, with additional insights on using ArcSight alongside Kafka and other security analytics tools.

Java programmers interacting with Elasticsearch have traditionally used either the REST API over HTTP or the internal Java API for node-to-node communication, with the latter offering potential performance advantages by bypassing the JSON parsing step required by REST. However, the Java API presents challenges such as frequent breaking changes due to internal class updates, backward compatibility issues, and dependency complexities, which complicate upgrades and security. In contrast, the REST API offers greater stability and can be upgraded independently of the Elasticsearch cluster, simplifying security and compatibility. To address these issues, Elasticsearch introduced a low-level Java REST Client in version 5.0.0, offering advantages like compatibility with any Elasticsearch version, load balancing, failover capabilities, and minimal dependencies. The plan is to develop a high-level Java REST client to further ease the transition from the Java API by reusing existing requests and responses while not fully implementing the existing Client interface. This shift marks a strategic move towards phasing out the Java API, enhancing user experience, and improving feature development and security for Java users of Elasticsearch.

Kibana's scripted fields allow users to dynamically create and manipulate data fields in Elasticsearch for enhanced data visualization and analysis, even if those fields were not originally defined in the schema. Initially limited by Lucene Expressions to numeric data, the introduction of the Painless scripting language in Elasticsearch 5.0 expanded the capability to handle a wider variety of data types, enabling more complex operations such as combining fields, performing calculations, and implementing conditional logic. Scripted fields are defined once in Kibana's Management section and can be reused across multiple visualizations, although they operate on single documents and are not suitable for cross-document calculations like time-series math. While powerful, scripted fields require careful management to avoid performance issues, and best practices suggest developing and testing them in non-production environments before deployment. Despite some limitations, such as the inability to use certain aggregations, scripted fields can significantly enhance data insights when used responsibly, with Painless providing robust support for string manipulation, regex matching, and date functions.

Elasticsearch 2.4.3, a bug fix release based on Lucene 5.5.2, has been announced, marking the latest update in the 2.x legacy series. This release addresses several issues, including the introduction of a HostFailureListener for the transport client to notify client code when a node disconnects and a bug fix that now allows the transport client to remove old nodes. Additionally, it resolves problems with cross-field multi-match queries not working with synonyms and function score queries not working with highlighting. Users are encouraged to upgrade if their systems are affected by these issues and are invited to provide feedback through Twitter or the Elasticsearch forum, with any problems reportable on the GitHub issues page.

Kibana 5.1.1 introduces several notable improvements and features, addressing performance and usability issues from previous versions. This release includes the addition of a tag cloud visualization, which graphically represents the importance of words in a body of text, and an Advanced View for X-Pack's monitoring feature that reinstates several charts to enhance data visibility. The update also integrates a UI for the profiler in the X-Pack basic install, aiding users in optimizing query performance by breaking down query timings. Performance enhancements include improved rendering, reduced CPU load from the progress indicator, and solutions to longstanding issues in the Discover app, such as handling indexes with numerous or overly large fields. Users can configure index patterns to exclude large _source fields, preventing Discover from failing on sizable data entries. The release promises an overall faster and more efficient user experience.

Elasticsearch 5.1.1, based on Lucene 6.3.0, has been released and is available for deployment on Elastic Cloud, addressing several enhancements and fixes from its predecessor, 5.1.0, which was skipped due to unreleased binaries mistakenly labeled under that version. Key updates include improvements in search functionalities, such as the introduction of the all_fields mode in queries, which advances towards eliminating the need for the _all field and optimizations in term queries to enhance performance and reduce garbage collection. Painless scripting has been improved with features like null-safe operators and negative offset addressing in arrays, while DNS lookups are now performed lazily and in parallel, bolstering reliability and flexibility. These enhancements aim to improve user experience, and users are encouraged to provide feedback through social media and GitHub.

Elastic has announced the public alpha release of Elastic Cloud Enterprise, a new product designed to enable users to deploy and manage Elasticsearch and Kibana clusters on their own infrastructure, whether on physical hardware or a hosting provider. This marks a significant milestone following substantial investment in developing a robust Elasticsearch and Kibana service on AWS and other public cloud platforms. Elastic Cloud Enterprise offers centralized management of multiple clusters, complete with features like provisioning, monitoring, scaling, and backup, thereby simplifying operational lifecycle management and enhancing hardware utilization. The product is especially beneficial for organizations needing to manage large deployments, enabling them to offer services like Logging as a Service by deploying multiple small nodes. The public alpha is intended for testing, and Elastic encourages feedback to refine the product before its general release, even incentivizing early testers with recognition and rewards.