October 2016 Summaries

Masquerading, a technique once used by the elite to hide their identities at dances, is now employed by cyber attackers to evade detection by blending into digital environments. This strategy involves mimicking common filenames and paths to appear legitimate, making it challenging for defenders to identify threats amidst vast data. The text outlines methods to hunt for such masquerading activities, including building an anchor list of legitimate filenames and comparing it against running processes. It also highlights the use of advanced tools like Endgame for automated detection, emphasizing the importance of routine updates and vigilance in maintaining effective cybersecurity defenses. While manual detection is possible, it can be labor-intensive and less scalable, making automated solutions more appealing for quick and efficient threat identification.

The release of Kibana 5.0.0 marks a significant upgrade in its design and functionality, requiring Elasticsearch 5.0.0 to operate. This new version introduces a refreshed color scheme, enhanced visualization capabilities, and improved user interface accessibility, including the elimination of unnecessary borders. Timelion, a time series data visualization tool, is now integrated into Kibana's core, allowing users to execute complex queries and generate visually appealing charts. The Console feature, formerly known as Sense, now simplifies the creation of custom Elasticsearch queries with features like autocomplete and retains previous requests. Users can also leverage the new Painless scripting language for scripted fields, offering a more secure alternative to Groovy. Additionally, the X-Pack extension provides built-in monitoring, authentication, and reporting capabilities. The upgrade process is straightforward for users of Kibana 4.6 and earlier versions, with various pre-release versions contributing to the refinement of Kibana 5.0 through community feedback and testing.

Elasticsearch 5.0.0, based on Lucene 6.2.0, marks a significant milestone in its development with the introduction of numerous enhancements and features aimed at improving performance, usability, and security. This version offers a remarkable increase in indexing throughput, thanks to improved numeric data structures and reduced contention in document updates. It also simplifies data ingestion through the introduction of the Ingest node, which incorporates popular Logstash filters directly into Elasticsearch. A new scripting language, Painless, is introduced, providing a secure and efficient alternative to Groovy. Additionally, new data structures such as Block K-D trees enhance numeric and geo-point indexing and querying, while improvements in search and aggregation processes boost performance and flexibility. With an emphasis on user-friendliness and resilience, Elasticsearch 5.0.0 introduces stricter validation of settings, deprecation logging, and a variety of tools to ease migration from previous versions. A new low-level Java REST client is also released, providing a stable alternative to the Java API. Overall, Elasticsearch 5.0.0 represents a comprehensive upgrade, offering improved speed, safety, and ease of use for users.

The Elastic Stack 5.0.0, released in October 2016, marks a significant milestone for Elastic, representing almost a year of development that included multiple alpha, beta, and release candidate stages. This release introduces the Elastic Stack, a unified suite of tools for data management and analysis, featuring enhancements such as the Ingest Node for data enrichment, Metricbeat for metrics collection, and Logstash monitoring APIs, along with the integration of Timelion in Kibana for data visualization. Additionally, the release highlights the new X-Pack, which consolidates previously individual features like security, monitoring, and reporting into a single package, simplifying the installation process and offering both free and commercial licensing options. The Elastic community played a vital role in shaping the release, filing numerous issues that informed development, and the Elastic Cloud now hosts the latest code for Elasticsearch and Kibana. The release also celebrates the community's engagement, with 75 million downloads to date, and invites users to a virtual event to explore these updates further.

X-Pack 5.0.0, released alongside Elastic Stack 5.0, consolidates and enhances the functionalities of several standalone plugins into a single extension that provides security, alerting, monitoring, reporting, and graph capabilities. This package simplifies installation and configuration processes, reducing them to just two commands while introducing default accounts for initial setups. X-Pack builds on the features of its predecessors, offering a management UI for security in Kibana, an improved alerting system with customizable notification conditions, and comprehensive monitoring for the entire Elastic Stack. It also facilitates the creation and sharing of Kibana visualizations and introduces a graph API and UI for exploring data relationships. While X-Pack features are part of subscription services, its monitoring capabilities are available for free with a Basic license, inviting users to explore its extensive functionalities.

Elasticsearch for Apache Hadoop (ES-Hadoop) 5.0.0 offers significant updates and enhancements that improve stability, introduce new features, and ensure compatibility with newer versions of integrations. Key updates include dropping support for older versions of Hive, Storm, and Spark to improve the codebase, while adding compatibility with Hive 1.0, Storm 1.x, and Spark 2.0. The HDFS Repository now integrates directly with Elasticsearch, eliminating the need for disabling the JVM SecurityManager. Additionally, the release introduces the Scroll Slicing functionality for increased parallel computing, native support for Spark Streaming to address connection resource limitations, and the ability to specify ingest pipelines and nodes to optimize data flow. The release also addresses various bugs, enhancing the overall performance and reliability of the software.

Logstash 5.0.0, a major release of the data processing tool, introduces significant enhancements, including improved self-service troubleshooting, performance upgrades, and enhanced logging capabilities. The new version offers a monitoring feature for real-time visibility into the Logstash pipeline, allowing users to query operational metrics via APIs modeled after Elasticsearch. Logging improvements include a switch to Log4j2 for more granular logging and dynamic log level adjustments without restarting Logstash. Performance has been boosted with a Java rewrite of the pipeline infrastructure, resulting in up to a 75% increase in event throughput. The release also features ease-of-use improvements, such as settings configuration through a single file, and updated release packages for consistency with other Elastic products. New plugins and integrations enhance compatibility with Elasticsearch 5.0.0 and Kafka 0.10, while the introduction of the dissect filter offers a simpler alternative for field extraction from structured log lines. The release underscores community contributions and aims to deliver a more efficient and user-friendly experience.

Kibana version 4.6.2 has been released to address a high severity cross-site scripting (XSS) vulnerability, identified as ESA-2016-07, which allows attackers to execute arbitrary JavaScript in the browsers of other users. Users operating Kibana versions 4.3 to 4.6 are advised to upgrade to version 4.6.2 immediately to mitigate this security risk, while version 4.1.11 remains unaffected. The updated version is readily available for download on the official website and through package managers such as apt and yum.

Apache Lucene, initially developed as a text search engine, has evolved significantly to accommodate numeric range filters, enhancing its functionality beyond text tokens. Originally, numeric values were treated as textual tokens with left-zero-padding to ensure correct sorting, but this method was inefficient and inflexible. Uwe Schindler's introduction of numeric tries in Lucene 2.9 marked a turning point, offering improved numeric support with a more user-friendly API and faster query performance by indexing numbers at multiple precision levels. The evolution continued with Lucene 6.0's block KD (BKD) tree data structure, initially implemented for geo-spatial searches but later recognized for its efficiency in one-dimensional numeric filtering. This advancement supports complex data types, such as IPv6 addresses, and significantly improves performance and index size, surpassing previous methods. Elasticsearch 5.0.0 leverages these advancements, showcasing the collaborative and iterative nature of open-source development in enhancing search capabilities.

Elastic{ON} 17, the third annual user conference for the Elastic Stack, is set to take place from March 7-9, 2017, in San Francisco at Pier 48, gathering 2,500 developers, architects, and IT professionals. This event will feature a variety of activities, including product roadmaps, deep dives, and networking opportunities, alongside unique experiences like the expanded Spotlight Theater and a diverse array of food trucks. New educational options are being introduced with pre-conference training workshops covering topics from beginner Elastic Stack usage to advanced Elasticsearch. The conference will also host the Elastic Cause Awards, recognizing projects that tackle real-world problems, and partner with Django Girls to support their workshop and initiatives. Attendees will have the opportunity to engage with the entire Elastic engineering team and meet key figures behind Elasticsearch, Kibana, and other components of the Elastic Stack.

Working at Elastic exemplifies a culture of freedom and trust, allowing employees to work remotely and independently, fostering a sense of belonging and teamwork. This approach contrasts with previous experiences where being physically present in an office was mandatory. At Elastic, the emphasis is on establishing trust and shared objectives, enabling employees to work effectively from anywhere in the world while feeling part of a supportive family-like team. This culture of freedom is complemented by the use of tools like Zoom to maintain connectivity and social interaction among globally distributed team members. The company has experienced significant growth, expanding its support team from 6 to 34 engineers, reflecting the success and increasing user base of its products. Employees are encouraged to explore the world, as demonstrated by experiences working across the globe, which enrich personal and professional lives. This sense of freedom and responsibility requires effective communication and self-management, making Elastic an attractive workplace for those passionate about open-source technology.

Elasticsearch 5.0.0-rc1, based on Lucene 6.2.0, has been released for testing, bringing developers closer to the general availability of version 5.0.0. This release introduces strict URL query-string parameter parsing to prevent issues with unknown or mistyped options by throwing exceptions, making problem identification and resolution more straightforward. The release is intended for testing purposes, and users are encouraged to provide feedback on any issues encountered. Additionally, the Elasticsearch Migration Helper is available to assist users in transitioning from versions 2.3.x/2.4.x to 5.0, offering tools like Cluster Checkup, Reindex Helper, and Deprecation Logging. The release notes for previous versions, including several alpha and beta releases, detail various enhancements such as indexing performance improvements and new features like Painless scripting and instant aggregations, which contribute to the system's resiliency and safety in production environments.

Kibana 5.0.0-rc1 has been released as a prelude to the upcoming general availability of Kibana 5.0, encouraging users to test and report bugs to ensure the final release is stable. This release candidate includes all features intended for the 5.0 version, although the CSV upload feature, initially introduced in an alpha version, has been removed due to quality concerns. Improvements have been made to visualization rendering performance by optimizing the rendering process, resulting in faster initial chart loads. Additionally, various fixes and enhancements have been included, such as more reliable console settings and improved handling of index patterns. Users are invited to download and test this release, providing feedback and reporting issues to aid in the refinement process.

Elasticsearch 5.x introduces ten new "bootstrap checks" that run during startup to identify configuration issues that could lead to significant failures under heavy use, particularly when the system is bound to a non-local IP address. These checks, designed to be minor annoyances, include a test for the inappropriate use of the "serial" garbage collector, which can cause nodes to temporarily leave and rejoin the cluster, and a check for low file descriptor limits, which can lead to failures in shard allocation. While the checks initially issue warnings, the Elasticsearch team emphasizes the importance of addressing these configurations to ensure stability and security, transitioning them to errors to compel users to fix the issues before deployment. This approach aims to prevent data loss and system instability in production environments by ensuring that "real" installations meet necessary configuration standards, particularly when setup involves non-local IP addresses indicative of production use.

The article by Marcelo Rodriguez provides a comprehensive guide to setting up and configuring Watcher, a flexible alerting tool within the Elasticsearch ecosystem, designed to trigger alerts based on specific conditions in streaming data. It details the installation process of the Watcher plugin, including necessary prerequisites such as the license plugin, and offers insights into configuring Watcher to send alerts via various platforms like email, Slack, and PagerDuty. The author explains the core components of a watch—schedules, inputs, conditions, and actions—through practical examples that demonstrate how to create basic alerts and more complex conditional alerts using Elasticsearch indices. The text emphasizes Watcher's versatility in monitoring various data sources and its capability to notify users of significant events, with references to further resources for advanced configurations and debugging.

The blog post by Pius Fung introduces the integration of Reporting and Watcher within Kibana, illustrating how users can generate and automatically send PDF reports based on specific conditions using Elastic Cloud. The process involves setting up Elasticsearch and Kibana, populating data, and securing it through the Shield for authentication. The example use case described involves sending bi-weekly PDF reports to a systems team manager if helpdesk tickets mentioning attachments are filed. The setup includes creating a dashboard in Kibana, installing necessary plugins, and configuring Watcher to send emails with the report attached. Fung explains how the system utilizes Watcher's email action to automate report generation and distribution, highlighting advancements since the original release and demonstrating the practical application of these tools in managing internal helpdesk tickets more efficiently.