April 2017 Summaries

Elastic has integrated Salesforce Community Cloud with Elastic Cloud to enhance customer support operations. The team moved their support portal from Zendesk to Salesforce, enabling the use of Elasticsearch to create a recommendations engine for knowledge base articles. This engine suggests content based on the case subject and the customer's platform, improving the support experience. The integration involved setting up a small Elastic Cloud cluster, and the framework developed is reusable for future applications. They plan to expand the integration by indexing more content sources and support interactions into Elasticsearch. Elastic invites engineers interested in this project to join their team.

Elasticsearch announced the release of versions 5.3.2 and 2.4.5, which are based on Lucene 6.4.2 and 5.5.4 respectively, and are available for deployment on Elastic Cloud. The 5.3.2 release addresses critical issues such as preventing node restart problems for stored template users introduced in version 5.2.0, and resolves edge cases related to a safety check for the Multi data path bug found in 5.3.0. Users are encouraged to upgrade to 5.3.2 to benefit from these bug fixes and share feedback on Twitter or the Elastic forum, with any issues being reportable on the GitHub issues page.

Paul Privateer, the founder of NoSchoolViolence.org, utilizes Elasticsearch to develop a tool called The Lantern, which aims to identify predictive behavioral patterns in adolescents to prevent school violence. Motivated by his experiences as a professor and the frequent claim that tragedies were unforeseeable, Privateer sought to identify patterns of behaviors such as slipping grades and antisocial tendencies that might serve as early warnings. Recognizing twelve risk factors acknowledged by major organizations like the CDC and the FBI, he and his team used Elasticsearch to manage vast amounts of data from police records and academic journals, allowing them to correlate specific behaviors with potential forms of violence. This approach not only provides a proactive resource for parents and educators but also aims to refine intervention programs as the technology develops, potentially expanding its application to health and academic performance issues.

Logstash is a versatile tool for ingesting and transforming event data, allowing users to stream data into Elasticsearch and visualize it with Kibana, while also supporting over 70 different outputs, including Amazon S3 for long-term archival. The S3 output feature enables users to store events in an S3 bucket for future retrieval or batch processing, with the option to use conditional logic to selectively route data streams to S3. Users can dynamically configure S3 prefixes using field references or event values, which is particularly beneficial for multi-tenant deployments. Recent updates to the S3 plugin include a move to asynchronous uploading, support for the latest AWS-SDK version, and the ability to dynamically choose the S3 destination structure. Data is buffered locally before being uploaded to S3, and the upload process is controlled by configurable rotation strategies based on file size and creation time. These enhancements provide users with greater control over their data archiving processes, making Logstash a practical tool for managing event streams efficiently.

Elastic announced a shift in its support model for Elastic Cloud Standard customers from a public forum to a ticket-based system, effective May 1, 2017, to improve service quality and customer experience. This change was driven by the need for ease of use, secure information sharing, and consistent, personalized support, allowing customers to receive dedicated assistance from a team of engineers via email. The new system also offers a better way to track customer history, aiding in more effective troubleshooting. Elastic expressed gratitude to its customers for their feedback, which informed this decision, and emphasized its commitment to continually improving services. Additionally, Elastic launched its cloud service on the AWS SaaS Marketplace, allowing for simplified billing through AWS.

Rashid Khan's blog post discusses how to deal with sparse time series data in Timelion, illustrating the challenge of visualizing data collected at one-minute intervals when a per-second rate is desired. The author explains how Timelion initially misrepresents such data by showing zeros for most seconds and how to correct this by using the .if() function to replace zeros with nulls. Khan warns against using linear interpolation, as it inaccurately inflates data, and instead advises distributing the total data evenly using .fit(scale) to maintain accuracy. The post concludes by demonstrating how to ensure a consistent per-second rate display with the .scale_interval(1s) function, highlighting how Timelion’s features can be applied to normalize sparse datasets and adapt them to various resolutions.

Elasticsearch's Cross-Cluster Search, introduced in version 5.3.0, enables users to perform searches across multiple clusters, overcoming the limitations of the previously used Tribe Node. Unlike the Tribe Node, which requires a separate node to join multiple clusters and manage cluster state updates, Cross-Cluster Search allows any node to act as the coordinating node without the need for additional nodes or complex configurations. This new functionality optimizes federated search by using the existing _search API, simplifying the process of merging and querying data across clusters. It addresses the Tribe Node's limitations, such as the inability to manage indices with the same name across different clusters and the requirement for node restarts when updating remote clusters. Furthermore, by managing memory consumption during search requests and aggregations, the new approach offers a more scalable and efficient solution, promising an improved user experience and setting the stage for future enhancements.

Elasticsearch 5.3.1, based on Lucene 6.4.2, has been released as a bugfix update and is available for deployment on Elastic Cloud. This release addresses several issues from version 5.3.0, including a bug related to the multi data path configuration that incorrectly added paths instead of overwriting them, and provides a check for residual data in default paths. It also introduces a safeguard against out-of-memory errors caused by misconfigured shingle and CJK token filters, which can generate excessive paths and consume heap space. Cross-cluster search now supports clusters with X-Pack Security enabled, and other improvements include a change in the Netty receive predictor size and a fix for issues with sliced scrolls and reindex-from-remote operations. Users are encouraged to upgrade and share feedback through social media or the forum, and any issues can be reported on GitHub.

Earlier this week, a security flaw was identified in Apache Log4j version 2, which could allow remote code execution by exploiting object deserialization within the JVM process. Although the Log4j plugin is included in a default Logstash installation, it is not enabled by default, and thus users not explicitly using it are unaffected. While current exploits do not affect default Logstash deployments, the Elastic Security Team acknowledges the inherent vulnerability due to Logstash's role as an endpoint for log data from various sources, which makes it impractical to secure completely. Consequently, Elastic has patched Log4j in Logstash and has deprecated the Log4j input, advising users to replace the Log4j's SocketAppender with Filebeat for safer log transport. This move aims to eliminate the security risks associated with object deserialization, and Log4j support will be removed in Logstash 6.0. Elastic extends thanks to Marcio Almeida de Macedo for bringing this issue to their attention.

Go applications can be effectively monitored using the Elastic Stack, specifically leveraging Metricbeat, a lightweight data shipper designed to track service and server metrics. Metricbeat, written in Go, provides modules for a variety of services and is extensible for additional ones. The Golang Module for Metricbeat, integrated into the Elastic Stack's master branch and set for release in version 6.0, allows for detailed analysis of Go's memory usage, garbage collection, and potential memory leaks via Kibana dashboards. By enabling Go's expvar service and configuring Metricbeat, users can monitor internal variables and statistics, exposing them as JSON data and allowing for visualization in Kibana. Additionally, Metricbeat can self-monitor its metrics, which can be analyzed using Timelion expressions to assess the stability of data exchanges with Elasticsearch. Users can begin utilizing this module ahead of its official release by accessing the Beats repository on GitHub.

Emily Mosher discusses how Daniel Myung and Bhasker Bokuri, scientific computing experts at Merck Pharmaceuticals, are utilizing the Elastic Stack to manage and analyze the vast and complex data generated by the Human Genome Project, aiming to improve the drug discovery process. By leveraging Elastic's capabilities, they can transform raw genomic data into a more manageable, searchable format, which allows researchers to better predict the efficacy of drugs in the early discovery phase, potentially reducing costs and increasing the success rate of new treatments. Despite the initial public expectation that the Human Genome Project would lead to immediate cures for diseases, the reality has been a prolonged process of data analysis and integration. Myung and Bokuri's approach, likened to interpreting weather data, involves harmonizing diverse genomic data sources into a universal coordinate system, enabling faster and more insightful searches that can identify promising drug targets, as illustrated by retrospective insights from the cholesterol drug Lipitor. While the database is still a work in progress, there is hope that incorporating machine learning and statistical methods will further enhance its utility for geneticists, ultimately aiming to raise the pharmaceutical industry's low success rate in developing safe and effective drugs.

In this blog post, CJ Cenizal discusses the Kibana team's process of componentizing their user interface using React, a JavaScript library for building UIs. The team found React's component-based architecture to be a natural fit for their goals, as it allows for encapsulation, making UI code more readable and maintainable while facilitating a faster development cycle. By using React, engineers can focus on integrating event handlers and business logic, rather than worrying about implementation details like CSS and markup. The post also highlights the importance of composition in component development, where React's design supports creating flexible and reusable components that adhere to the single responsibility principle. The team uses tools like AirBnB's Enzyme and Facebook's Jest for unit testing, ensuring that components are reliable and consistent. The article concludes by inviting readers to follow their transition to React through GitHub and participate in an open-source React Design Workshop, with a teaser about applying componentization to the design process in the next installment.

The blog post discusses how to utilize the Elastic Stack for transforming log files into valuable insights through GeoIP translations, which convert IP addresses or hostnames into geographic coordinates. The process involves using the Ingest API and Logstash to extract and enhance log data with geo-information, enabling visualization on maps using Kibana. The Ingest API, added in Elasticsearch 5.0, allows document processing with plugins, such as the geo-ip plugin, while Logstash provides a similar capability with its geoip filter. The post also highlights the importance of correctly configuring templates and mappings to ensure geoip fields are treated as geo_point data types in Elasticsearch. Additionally, it addresses common troubleshooting issues, such as field mapping errors and geolocation inaccuracies, and provides tips for overcoming challenges related to map visualization in Kibana. By following the outlined steps, users can effectively plot geographic data from logs, enhancing decision-making for various applications like marketing and network capacity planning.

Elastic's Elasticsearch Service and Amazon's Elasticsearch Service are distinct offerings, with Elastic's version running on Elastic Cloud and integrating with AWS, Google Cloud Platform, and Microsoft Azure. Elastic Cloud offers hosted Elasticsearch, Kibana, and X-Pack services, which include advanced features like security, alerting, monitoring, and reporting, that are not available with Amazon's version. Elastic emphasizes the advantages of their service, such as same-day updates, robust security patches, and increased flexibility in deployment and management. They offer a competitive pricing model and claim superior technical support and operational excellence due to their extensive experience with the Elastic Stack. Elastic Cloud supports a variety of customers and provides a Service Level Agreement (SLA) focusing on cluster-level availability to ensure uptime for mission-critical applications.

In a blog post from Elastic{ON} 2017, the authors discuss the creation of an operational analytics demo using the Elastic Stack to enhance the attendee experience and improve future events. The demo utilized real-time data from the conference, such as wireless network traffic to track crowd movement and sentiment feedback from attendees on session quality. The data was ingested using Elasticsearch, combining inputs from SNMP polling, Packetbeat, and Logstash pipelines, to provide insights into session popularity and network usage. The blog highlights the challenges of capturing and processing this data, particularly concerning privacy concerns and the accuracy of location-based analytics. The authors detail the technical setup, including the use of Ruckus wireless access points and a Juniper switch, and how they enriched the data with session topics and geo-information. The ultimate goal was to identify trends and improve event planning by correlating network traffic with session engagement, with future blog posts promising to explore data visualization techniques using Kibana.

In Elasticsearch 5.3.0, a bug affecting users who configure a custom data path could lead to data loss if not managed correctly. The issue arises when the `path.data` setting in the `elasticsearch.yml` file is specified as an array, causing it to merge with the `default.path.data` setting instead of replacing it. This can result in multiple data paths being used, potentially leading to shard allocation in unexpected locations and startup failures for additional nodes due to path locking conflicts. To mitigate the bug, users should adjust their configuration to use a comma-separated string for `path.data`, ensuring it overwrites `default.path.data`. Additionally, data from the default path must be moved to the other specified paths before restarting nodes, with the process verified through cluster health checks. Future versions, starting from Elasticsearch 5.3.1, will include a fix for this issue and will prevent the node from starting if shard data is found in the `default.path.data` directory, prompting users to clear or relocate the data to avoid complications.

Managing software assets effectively becomes challenging for large organizations with extensive IT infrastructures, especially during audits, which can reveal discrepancies in software usage and licensing. Utilizing Elasticsearch and Software Defined Networking telemetry, network traffic analysis provides a solution by tracking software activity through server communication patterns. For instance, applications like Adobe Photoshop and Illustrator generate identifiable network traffic, allowing the calculation of user numbers and usage times by analyzing data such as the Server Name Indication and timestamps. The process involves filtering relevant data, using date histograms for time-based aggregation, and employing cardinality and scripted metric aggregations to determine user counts and usage durations. This method leverages network data to produce insightful analytics, which can be visualized using tools like Chart.js, highlighting the potential of combining network functions with the Elastic Stack for robust IT asset management.

Recent improvements in Elasticsearch have enhanced the efficiency of range queries, particularly when used in conjunctions with other queries. Traditionally, range queries faced challenges because they efficiently iterated over matches but struggled with verifying individual document matches due to the way numerics are indexed. By leveraging doc values, which provide a per-field lookup structure, Elasticsearch can now verify document matches more effectively. The introduction of a new mechanism, IndexOrDocValuesQuery, allows Elasticsearch to choose between using index structures or doc values based on whether the operation requires sequential access or random access. Benchmarks show significant performance improvements, especially for range queries intersecting with selective term queries, demonstrating a 30x speedup in some cases. Although the strategy is not flawless, as query planning is complex, these changes promise better query execution and have potential applications beyond range queries to include geo bounding-box and geo-distance queries. The enhancements are set to be included in Lucene 6.5 and Elasticsearch 5.4, suggesting future improvements for other query types that currently evaluate against the entire index.

Curator 5 has been released, introducing several new features and changes tailored for Elasticsearch 5.x users while maintaining the previous configuration format for ease of upgrade. Key updates include the addition of a powerful reindex action allowing users to reindex locally or remotely with various filters, alongside a new rollover function that facilitates index management based on specified conditions. Curator 5 also introduces date math in index creation for future timestamped indices, an unset shard routing allocation feature, and a period filter for selecting time-based index blocks. Enhancements have been made to the wait-for-completion functionality, reducing client connection binding for long-running actions, although forceMerge remains an exception due to its inherent characteristics in Elasticsearch. Overall, Curator 5 offers a robust set of tools for efficient index management and improved user experience.

The Nature Conservancy (TNC), a global nonprofit focused on conservation, faced challenges in securing its vast network spread across 70 countries and numerous remote and low-connectivity locations. IT and security experts Nick Waringa and Daniel Shirer utilized Elastic Stack, X-Pack, and Elastic support services to enhance their network security and data analysis capabilities. Their approach included deploying threat intelligence and network monitoring tools like Snort and Bro IDS, and using Elastic tools like Filebeat, Metricbeat, and Logstash for data processing, with Elasticsearch and Kibana for visualization. This setup enabled them to gain comprehensive insights into network behavior and foster stronger collaboration with TNC's operations team, opening new avenues for data-driven decision-making and potential expansions into areas like micro-level climate change monitoring. Their innovative work in deploying consumer-grade hardware for security infrastructure earned them a CSO50 award for demonstrating significant business value and thought leadership.

In Part 4 of the blog series on integrating Elasticsearch with ArcSight SIEM, the focus is on setting up alert systems for detecting successful brute force SSH login attempts using Elasticsearch's X-Pack. The post details a complex process of identifying suspicious login patterns, specifically multiple failed login attempts followed by a successful one within a defined time window, using Elasticsearch aggregations and Painless scripting. The watch, or alert, is configured to assess login data, identify potential brute force attacks, and prevent duplicate alerts by indexing detected threats. The document also outlines steps for transforming and logging alerts and suggests enhancements like adding server-specific analysis and integrating with notification systems for better real-time alerting. This setup highlights the benefits of automated anomaly detection through machine learning, which simplifies the process by reducing the need for complex rule definitions.

Kibana is expanding its visualization capabilities, introducing new features such as heatmap and point series charts to enhance data representation. Heatmaps, launched in version 5.2, allow users to plot individual bucket values as colors, making them ideal for visualizing metrics over time, like website visitors segmented by operating systems. Users can customize heatmaps with various color schemes and labels. The upcoming 5.4 release will enhance point series charts, including line, area, and bar charts, with multiple Y-axis support, mixed chart types, and the addition of horizontal bar charts. These improvements aim to provide more flexibility and precision in data visualization, allowing users to easily switch chart types and manipulate axis positions for optimal clarity. The enhancements are available for preview in master and nightly snapshot builds of Kibana, and feedback is encouraged through GitHub and forums.