August 2015 Summaries

Elastic's blog post discusses the differences between the Watcher and Percolator APIs within the Elastic product stack, focusing on their distinct use cases and functionalities. While both are used for matching documents against queries, Percolator is designed to ascertain which queries a specific document would match in real-time, making it suitable for applications needing immediate alerting on new documents. In contrast, Watcher periodically executes pre-defined queries against a document corpus, allowing for trend analysis and actions based on changes over time, such as monitoring seasonal trends or periodic data checks. The post also touches on the scalability of both tools, explaining that Percolator scales linearly with the number of queries and excels in scenarios with high query cardinality and low document throughput. The Watcher, on the other hand, is advantageous for scenarios that require periodic monitoring and can handle a larger document corpus. The blog also mentions future enhancements to Watcher, including the potential for real-time alerts and integration with Percolator results.

Kibana 4.2.0-beta1 has been released to complement the Elasticsearch 2.0.0-beta1, providing new features and bug fixes. Notable updates include persistent colors that ensure consistent color representation on dashboards, a server status page that appears when connectivity issues arise, and a feature that allows filtering through a visualization's legend. Additionally, Olson timezone support in Elasticsearch 2.0 enhances the accuracy of event timing across different time zones. Structural improvements involve updating libraries, ensuring consistent builds through "shrink-wrapped" dependencies, and conducting unit tests in various browsers to better reflect real-world usage. Users are encouraged to engage with the community on the Discuss Kibana forums and report any bugs on GitHub.

In late August 2015, Elastic announced various global meetups, highlighting their community's ongoing engagement. The events included a Logstash Night Meetup in Silicon Valley and discussions on Elasticsearch in Grimstad, Norway, and Amsterdam, focusing on its resilience. Elastic encouraged community members to host or speak at future events, offering support and promotional materials to facilitate participation and connection within the community.

Elasticsearch's _ttl (Time to Live) mapping allows for setting expiration times on documents, leading to their automatic deletion after the specified period. However, the _ttl feature is deprecated, and its use is discouraged due to the inefficiency of the deletion process, where expired documents are marked for deletion but remain in the index until segment merges occur, causing performance issues. Instead, time-based indices are recommended, which involve using expiration fields for filtering and index aliases to improve query performance and reduce overhead. These indices allow for more efficient management by creating new indices based on time intervals and deleting entire indices when they are no longer needed, rather than handling numerous document deletions. However, certain data models, such as those requiring parent/child relationships, may not be suitable for time-based indices, and in such cases, alternative solutions or application-level TTL management should be considered. Additionally, the presence of Elasticsearch's Shield can create challenges in deleting documents with _ttl due to permission issues, though this does not affect time-based indices.

Elasticsearch 2.0.0-beta1, based on Lucene 5.2.1, has been released for testing, featuring over 2,500 pull requests from 469 contributors, with 850 new additions specific to this version. This beta includes significant updates such as pipeline aggregations, improved store compression, optimized heap usage, structured exceptions, and refined query execution. It also introduces changes aimed at enhancing stability and performance, like atomic transactions and improved type mappings. Core and commercial plugins, such as Shield and Watcher, are synchronized with Elasticsearch's versioning and release schedule. A Migration Plugin has been introduced to aid users in preparing for the transition, by identifying potential issues like incompatible mappings. A known issue exists concerning fields of type IP, which will be addressed in a future update. Testers are encouraged to provide feedback to expedite the general availability release.

Elasticsearch is a versatile tool originally developed as a search engine but has evolved to serve a wide range of purposes, including assisting NASA with Mars missions, helping Opower reduce global energy consumption, enabling Wikimedia to make knowledge accessible, and aiding Yelp in finding dining locations. This adaptable platform allows users to interpret diverse data sets, from server logs to literary works, by connecting and analyzing data points. Elastic{ON} 15 captured various perspectives on Elasticsearch's functionality, showcasing its wide-ranging applicability and prompting users to share their descriptions on social media with the hashtag #elastic. The event, which took place in an improvisational setting, highlighted the community's engagement and anticipation for future gatherings.

The text is a brief update from Elastic about their upcoming meetups taking place in various regions around the world during the week of August 24, 2015. It highlights events in North and South America, Europe, and the Asia Pacific, specifically mentioning meetups in Philadelphia, Los Angeles, Frankfurt, Berlin, and Adelaide. The message, shared by Livia Froelicher, encourages community engagement by inviting individuals to host meetups or give talks related to Elastic's products, such as Beats, Elasticsearch, Logstash, or Kibana, with an offer of support and promotional items.

Kibana 4.1 introduced the concept of field formatters, which allow users to visually transform data fields without altering the underlying data. This feature is particularly useful for displaying information in a more comprehensible way, and the blog post provides a detailed guide on creating a custom field formatter. The process involves understanding the field formatter API, creating a basic formatter to highlight occurrences of the word "error" by either wrapping it in a mark element or converting it to uppercase, and registering it in the application. The formatter is further enhanced to allow pattern-based highlighting by using regular expressions, enabling users to define custom patterns in an editor interface. Additionally, the post discusses the importance of security measures like escaping values to prevent cross-site scripting attacks and mentions future plans to simplify the registration process by providing this functionality as a plugin. Kibana's field formatter API is designed to be flexible, permitting users to extend its capabilities with custom solutions tailored to their specific needs.

Logstash versions 1.5.4 and 1.4.5 have been released, addressing important security vulnerabilities and adding enhancements. The update primarily fixes an SSL/TLS certificate validation issue in Logstash Forwarder and a man-in-the-middle vulnerability in Lumberjack output, assigned CVE-2015-5619. Enhancements include the addition of new Elasticsearch output features, allowing document updates and upserts if documents do not exist, thanks to contributions from David Chauviere. Bug fixes include improvements to SSL handling, suppression of deprecation warnings, and better management of congestion scenarios in Lumberjack output. Users are encouraged to upgrade immediately and provide feedback through social media or GitHub.

With the release of Elasticsearch 2.0.0-beta1, the Delete by Query API has been transitioned from a core feature to a plugin due to significant issues with its previous implementation that included unexpected visibility of deletes, segment explosion, merge storms, and risks of data corruption due to inconsistent replicas. The new plugin addresses these problems by using a safer method involving scan and scroll requests to find document IDs before deleting them via the bulk indexing API, although this method is slower. The decision to shift to a plugin was made to maintain a reliable and minimal core, as the previous Delete by Query feature, despite its speed and popularity, posed risks that couldn't be ignored. Additionally, deleting a type's mappings has been removed due to potential index corruption risks, encouraging users to adopt alternative strategies such as using separate indices.

Natural Language Processing (NLP) is being adapted by data scientists at Endgame for security purposes, specifically to improve the detection and analysis of malicious code through a framework called Malicious Language Processing. This approach leverages traditional NLP techniques, such as tokenization and semantic network analysis, to parse and identify patterns in binary code, similar to how human language text is analyzed. The process involves static and dynamic analysis to build a comprehensive dataset that is then used to automate the identification of malicious elements within benign code. The initiative aims to address large-scale security challenges by enhancing techniques like Domain Generation Algorithm classification, source code vulnerability analysis, phishing detection, and malware family analysis. Although still in its early stages, the initiative seeks to develop tools like a malicious stop word list and an anomaly detector for more efficient malware behavior analysis, with the ultimate goal of understanding suspicious binaries without human intervention.

"Where in the World is Elastic?" is an update from August 17, 2015, detailing various upcoming events and meetups related to Elastic, a company specializing in search and data analytics solutions. The post highlights activities in Europe, such as Isabel Drost Fromm's presentation at FrOSCon in Germany and the Elastic bouncy castle, as well as meetups in North/South America and the Asia Pacific, including locations like Los Angeles, Montevideo, Kansas City, Singapore, and Sydney. The Elastic Team also invites readers to host meetups or give talks about their products, offering support and promotional items to interested parties.

Zachary Tong's article explores how moving averages, specifically through the use of pipeline aggregations in Elasticsearch, can be utilized to create dynamic control charts capable of managing complex data trends. Initially introduced with a simple dataset, the article demonstrates the adaptability of control charts to handle more challenging data scenarios, such as linear and cyclic trends. For linear trends, the dynamic nature of moving averages allows the control chart to adjust without manual modifications. In cyclic data scenarios, the article recommends using the Holt-Winters model to better account for seasonality, addressing the lag problem inherent in simpler models like EWMA. Additionally, the article explains how to configure a Watcher alert in Elasticsearch to automatically detect and notify users of data spikes, enhancing the practical application of these statistical techniques in real-time data monitoring. The piece underscores the power and versatility of moving averages not only for smoothing data but also for detecting anomalies, forecasting, and integrating alert systems directly within Elasticsearch.

The text, titled "Where in the World is Elastic?" from August 10, 2015, by Livia Froelicher, highlights upcoming meetups organized by Elastic across North America and Europe during the summer. Scheduled events for the week include an Advanced Apache Spark Meetup in San Francisco on August 10, an Elastic Silicon Valley Meetup in Mountain View on August 12, a Rubyjax Meetup in Jacksonville-Beach on August 13, and a Pygrunn Meetup in Groningen, Netherlands, on August 11. The message encourages readers to stay updated for future events and invites those interested in hosting a meetup or speaking about Elastic's products like Beats, Elasticsearch, Logstash, or Kibana to reach out for support and promotional materials.

On August 10, 2015, Jay Modi announced the release of Shield 1.3.2, which primarily addresses several bug fixes, including improvements in forwarding audit events to remote clusters and preventing node startup failures due to LDAP connection issues. Users are encouraged to upgrade to this version to benefit from the fixes, with detailed upgrade instructions available for those updating from previous versions. New users can download the latest version directly.

The blog post by Tal Levy outlines how to use Logstash with the JDBC input plugin to import and manage data from various databases, like PostgreSQL, in Elasticsearch. It begins by explaining the ease of installing and configuring the JDBC plugin with compatible drivers from popular databases such as Oracle, PostgreSQL, and MySQL. The post provides practical examples, including a step-by-step guide for setting up a database and using Logstash to query and output data into Elasticsearch, ensuring synchronization to avoid duplicates. A demonstration using MusicBrainz showcases the capability to index complex data sets into Elasticsearch, allowing for advanced data exploration and visualization with tools like Kibana. The article encourages further exploration of Logstash's features and invites feedback and bug reports from users.

In the blog post "Where in the World is Elastic?" by Livia Froelicher, readers are provided with an overview of upcoming events and meetups related to Elastic, taking place globally from August 1-9, 2015. Highlights include several significant gatherings in Las Vegas, such as Blackhat, where a community speaker will discuss ElastAlert, and DEF CON 23, featuring Rashid Khan for discussions about the ELK stack. Additional meetups are scheduled across various locations, including Boston, Mexico, Toronto, Montevideo, Brisbane, and Sydney, focusing on topics like Elasticsearch, Hadoop, and DevOps. The Elastic Team also encourages individuals interested in hosting meetups or presenting on Elastic-related topics, offering support and promotional items.

Control charts, developed by Dr. Walter Shewhart in the 1920s, are utilized in manufacturing and business processes to detect when a process goes "out of control" by identifying deviations from the mean beyond normal variation. These charts are simple and effective tools for anomaly detection, and their modern implementation can be enhanced using Elasticsearch's pipeline aggregations. The text demonstrates how to smooth data using moving averages, specifically an Exponentially-Weighted Moving Average (EWMA), to better visualize trends and detect anomalies, such as a sudden spike in data. By incorporating a moving average on both the mean and standard deviation, a control chart can dynamically set an "upper control limit," allowing real-time detection of outliers that could indicate a problem, such as in the hypothetical scenario of monitoring a nuclear reactor's coolant temperature. The text also hints at further applications of control charts for more complex data patterns, such as linear trends and cyclic behavior, and suggests future integration with Watcher for automated alerts.