February 2016 Summaries

FireEye, a security company providing real-time threat protection to enterprises and governments, utilizes Elasticsearch in its Threat Analytics Platform (TAP) to guard against cyber-attacks without relying on signatures. Elasticsearch's evolution has been closely monitored by FireEye's engineering and operations teams, with milestones marked by Elastic{ON} conferences, where Elastic's company-wide presence allows for direct insights into product vision and improvements. At the 2016 conference, FireEye not only participated as attendees but also shared their experiences, discussing their objectives and exploring Elastic's roadmap, particularly in analytics. This engagement facilitated valuable exchanges with attendees about the practicalities and challenges of deploying Elasticsearch. Chris Rimondi, who leads the Site Reliability Engineering team at FireEye, highlights the significance of these interactions in learning and sharing practical deployment experiences, underscoring FireEye's commitment to leveraging Elasticsearch for indexing vast amounts of data to enhance security analytics for their enterprise customers.

Nik Everett's blog post provides a detailed guide on loading Wikipedia's search index for testing purposes using Elastic Stack version 6.3.1. The process involves downloading a search index dump from the Wikimedia Foundation, which publicly shares data for safe access, and then setting up Elasticsearch with necessary plugins such as analysis-icu and a Wikimedia-specific similarity plugin. Everett uses English Wikiquote as an example, illustrating the steps to prepare the index and load it into Elasticsearch, including creating a directory of smaller file chunks for efficient bulk loading. The procedure emphasizes the importance of monitoring progress and suggests optimizing for performance, although this might not fully replicate a dynamic production environment. The blog post concludes by encouraging users to experiment with queries on the loaded index, while also noting that larger wikis like English Wikipedia require more storage and time, reflecting on the practical challenges of handling large data sets on different storage media.

Amidst the vibrant atmosphere of San Francisco's Exploratorium during the Elastic{ON} 16 kickoff party, attendees were humorously asked to imagine a life without Elasticsearch, a tool integral to many of their daily operations. Participants expressed that without Elasticsearch, their lives would be significantly disrupted, with some joking about the tears and slower pace they would experience. This lighthearted inquiry was captured in a short video at the Elasticsearch user conference, showcasing the deep reliance and affection users have for the technology.

Logstash version 2.2.2 has been released with a crucial security bug fix addressing a vulnerability present in version 2.2.1, which made it susceptible to a man-in-the-middle attack when used with Elasticsearch output due to the inadvertent disabling of SSL/TLS configuration by default. This vulnerability allowed unauthorized access to payload data transmitted via HTTP during the initial handshake. Users are encouraged to upgrade to version 2.2.2 to resolve this issue, but those unable to upgrade immediately can mitigate the risk by using the "https" prefix in their host configuration and then restarting Logstash.

Elasticsearch can be effectively utilized for network security monitoring, specifically for detecting TCP host portscans, by leveraging its speed, scalability, and flexibility. The process involves capturing network traffic data using tools like tcpdump, structuring event data with Logstash, and indexing it in Elasticsearch. Aggregations within Elasticsearch help detect suspicious activity, such as a high number of unique port connections in a short timeframe, indicative of a portscan. To automate alerts, the Watcher feature can be configured to send email notifications when predefined conditions are met, such as detecting more than 50 unique port connections from a source to a target within 30 seconds. This approach demonstrates how Elasticsearch's capabilities can be adapted for robust security monitoring and proactive threat detection in large organizations.

Elastic has introduced Elastic Cloud and Elastic Cloud Enterprise, evolving from the acquisition of Found to provide a comprehensive Elasticsearch as a service offering. The Elastic Cloud includes Elasticsearch and a free Kibana instance, with premium features like security, monitoring, and alerting integrated through X-Pack. It allows users to set up clusters quickly with a 14-day free trial and offers seamless updates and tight integration with the Elastic Stack. Elastic Cloud Enterprise extends this by offering the same technology for installation on users' hardware, enabling centralized management of multiple deployments across teams or geographies. This flexibility allows organizations to either consume Elastic Cloud as a service or install it internally to manage Elasticsearch clusters, reflecting Elastic's commitment to listening to its community and offering scalable solutions.

Monitoring DNS traffic can serve as an indicator of compromise (IOC) in network security, especially in detecting data exfiltration via DNS tunnels, a method often used by malware to bypass corporate firewalls. By utilizing Packetbeat, an open-source packet analyzer, DNS requests and responses can be indexed into Elasticsearch, allowing for the analysis and aggregation of data. The setup involves configuring Packetbeat to observe DNS traffic and using Watcher, part of Elastic's X-Pack, to generate alerts based on specific conditions, such as a high number of unique hostnames associated with a domain, which may indicate a DNS tunnel. The process includes setting up triggers and conditions for alerts, which are executed through actions like sending emails or logging messages when suspicious activity is detected. Tuning involves adjusting variables such as the time window and hostname threshold to balance sensitivity and minimize false positives. This approach underscores the importance of a layered defense strategy to enhance network security.

LotaData is leveraging geo-temporal intelligence to provide businesses and brands with detailed neighborhood insights that can enhance marketing strategies and improve audience targeting. By analyzing data across multiple location-based datasets, LotaData constructs neighborhood profiles from hundreds of geo-temporal variables such as local demographics, events, businesses, and political leanings. These profiles, accessible via APIs, allow marketers to execute hyper-targeted campaigns by understanding the unique characteristics of each neighborhood. The platform is powered by Elasticsearch to ensure scalability and performance, enabling complex queries for precise market insights. Co-founder Apu Kumar, with extensive experience in technology, leads LotaData in transforming neighborhood data into actionable intelligence, helping brands like Campari America and Pinkberry optimize their local presence and campaigns.

Apache Lucene 6.0 introduces a new feature called dimensional points, utilizing the k-d tree geo-spatial data structure for efficient single- and multi-dimensional numeric range and geo-spatial point-in-shape filtering. This feature replaces deprecated numeric fields and offers improved performance and versatility, supporting up to 8 dimensions and 16 bytes per dimension. The block k-d tree variant is designed for efficient I/O, storing most of its data structure in on-disk blocks with a small in-heap binary tree to locate these blocks during searches. At index time, it recursively partitions N-dimensional points into smaller cells, and at search time, it efficiently tests query shapes against these cells. Dimensional points promise significant improvements in index size and search time efficiency compared to legacy systems, and although currently limited to single points, future enhancements may include indexing shapes with R-Trees. While not yet officially released, this feature offers exciting possibilities for combining geo-spatial and other dimensional data for advanced querying and filtering.

Elastic is hosting its largest annual event, Elastic{ON}16, from February 17-19 in North America, featuring three days of presentations, live demos, and discussions by Elastic developers and guest speakers. In addition to this major conference, several local meetups are scheduled across Europe and North America, focusing on various topics related to Elastic products and technologies. Highlights include discussions on Elasticsearch integrations, data science applications, and specific implementations like the elasticsearch-spark connector and building search applications with AngularJS. Attendees are encouraged to follow updates on Twitter and engage with the Elastic community through these events.

The Elastic Training experience described by Jason Dickson highlights the engaging and inclusive nature of Elastic's Kibana 4 Workshop, where participants, regardless of their technical background, can gain a thorough understanding of the Elastic Stack and its components such as Elasticsearch, Logstash, and Kibana. Conducted by knowledgeable instructors Kurt Hurtado and Shelby Sturgis, the course caters to IT professionals and novices alike, offering insights into data visualization, event data management, and practical applications. The workshop combines lectures with hands-on lab activities, allowing attendees to explore features of Kibana and Elasticsearch through real-world datasets. The instructors' approachability and willingness to address individual queries, as well as the course's balance between introductory and technical content, contribute to a highly interactive and educational experience. Jason shares his personal journey of overcoming initial apprehension and achieving a deeper appreciation for Kibana's capabilities, underscoring the value of Elastic's official training in equipping participants with essential skills and knowledge.

The Logstash Lines blog post from February 2016 announces the addition of a dynamic configuration reloading feature, allowing Logstash to update configurations without a process restart by using a new command option. This update, included in the 2.2 release and soon to be backported to version 2.3, is part of a broader release event that also includes version 2.1.2, featuring a new pipeline architecture. While a user-reported bug affecting certain filters has been addressed with a targeted 2.2.1 release, the Logstash team is also working on an API to expose internal metrics with improved filtering capabilities, requiring Java 8 for version 3.0.0. Additionally, enhancements such as a more efficient JSON Lines codec and considerations for using stored procedures in the JDBC plugin are underway. These developments and more will be discussed at the upcoming Elastic{ON} '16 conference in San Francisco.

NEST 2.0, the .NET client for Elasticsearch, has been released with significant updates and improvements since version 1.7.1, incorporating over 1000 commits. The developers restructured the internal architecture, replacing the Task Parallel Library with async/await for more efficient exception handling and providing an extensive audit trail. The test framework has been overhauled to streamline testing through a single project that supports both unit and integration tests, along with a Virtual Cluster for simulating various client scenarios. Exception handling has been simplified to three main types, enhancing user understanding of error origins. NEST 2.0 introduces numerous breaking changes, aligning with Elasticsearch 2.0's features, such as merging query and filter constructs and refining attribute-based mapping for cleaner API usage. The update also embraces new features from C# 6 and provides compatibility with DNX through a new dotnet5.1 version, while actively preparing for future .NET developments.

In early February 2016, Elastic announced a series of global events and meetups highlighting their technologies such as Elasticsearch, Logstash, and the Elastic Stack. Key events included a talk at DrupalGov Canberra in Australia about integrating Drupal with big data, and a series of meetups across Europe, including discussions on Elasticsearch upgrades and shard allocation in various cities like Geneva and Amsterdam. The announcement also highlighted events in South America and Asia, such as the PHPSC Floripa Meetup in Brazil and the Pune Elastic Meetup in India, where attendees could learn about Logstash's installation and configuration. Elastic encouraged community engagement by inviting individuals to host meetups and offer talks, providing support and promotional items as incentives.

Dimitrios Liappis details a process for creating a custom Amazon Machine Image (AMI) from a virtualization source such as a VirtualBox .vmdk file, which can be particularly useful when existing public AMIs do not meet specific needs. The procedure involves configuring a virtual machine with necessary tools like cloud-init and uploading the VM image to an S3 bucket for conversion using AWS ImportImage. Essential prerequisites include defining an S3 bucket, setting appropriate AWS roles and policies, and ensuring fast upload bandwidth. Liappis also introduces a tool called amiimport.py to automate this process, particularly for Vagrant boxes, by handling tasks such as uploading images and creating AMIs across multiple regions. The guide emphasizes security considerations, such as changing default passwords in Vagrant boxes, and acknowledges the challenges of resizing root partitions, suggesting that the automation tool could simplify the conversion process.

In the February 8, 2016 edition of "This Week in Elasticsearch and Apache Lucene," key updates include the release of Elasticsearch 2.2.0, featuring a query profiler and enhanced geopoint fields, alongside various improvements in the Elasticsearch core, such as better exception handling and the addition of new processors for ingest pipelines. The article also discusses ongoing developments like the reindex API's background operation capability and preparations for the Apache Lucene 5.5.0 release, highlighting the move of postings-based geo point queries to the spatial module for improved performance. Additionally, it outlines various testing and refactoring efforts aimed at enhancing both Elasticsearch and Apache Lucene, such as improving geo math calculations and introducing a utility class for debugging with TokenFilters.

Elasticsearch 2.2.0 has been released, featuring significant improvements such as a new query profiler and enhanced geo-point fields, both of which aim to boost performance and user insight into query operations. The update builds on Lucene 5.4.1 and includes tighter security measures, such as modularizing scripting languages to minimize security risks, and addresses a bug that caused slow shard recovery in the previous version 2.1.0. Accompanying Elasticsearch 2.2.0, there are updates to related tools like Kibana, Logstash, Beats, and Elasticsearch-Hadoop. Additionally, bug fix releases for versions 2.1.2 and 1.7.5 are available, and users are encouraged to upgrade to the latest versions to benefit from these enhancements.

Elasticsearch for Apache Hadoop (ES-Hadoop) has released versions 2.2.0 and 2.1.3, bringing significant updates and bug fixes to improve compatibility and performance with Elasticsearch and Apache Spark. Version 2.2.0 introduces compatibility with Elasticsearch 2.x while maintaining backward compatibility with 1.x, upgraded geo support, and enhanced network capabilities for various topologies. It also includes better runtime diagnostics to prevent misconfigurations, extended configuration options for multi-dimensional fields, and YARN module enhancements. The update tracks the latest versions of Apache Spark, leveraging new features to enhance performance and maintain backward compatibility, while repository HDFS is being overhauled for security improvements. Meanwhile, ES-Hadoop 2.1.3 serves as the final maintenance release for the 2.1.x line, offering backported bug fixes for users with conservative upgrade paths. Feedback is encouraged through GitHub, Twitter, forums, and IRC channels.

Logstash 2.2.0 and 2.1.2 were released as part of an extensive update to the Elastic Stack, featuring significant enhancements to Logstash’s internal architecture, notably with the introduction of the "Next Generation Pipeline" (NG pipeline). This new pipeline processes events in micro-batches, increasing performance by up to 28% compared to the previous version, and changes how the filter and output stages operate by running them in sequence in each worker thread. This version simplifies the management of output performance with automatic scaling and encourages plugin authors to use the multi_receive interface to handle events, which improves efficiency and moves closer to achieving event persistence. The Elasticsearch output bundled with this release has been optimized for the NG pipeline with improved retry handling and new features, such as support for scripts to update documents, aligning with the shift from node to HTTP protocol for easier operational management. Additionally, various bugs have been fixed, enhancing the functionality of file input, Kafka, and HTTP components, demonstrating the continued dedication to refining the user experience and system performance.

Logstash is undergoing significant changes to enhance its robustness and feature set, including the deprecation of certain legacy components like the node protocol for Elasticsearch output and the multiline filter. The node protocol, known for its complexity and maintenance challenges, is being replaced by the more efficient and easier-to-use HTTP protocol, which will become the default in Logstash 2.0. This shift is recommended before the next major release, especially since the node protocol is incompatible with Shield-protected Elasticsearch clusters. Meanwhile, the multiline filter is being deprecated in favor of the multiline codec, which offers improved performance and bug fixes, with migration encouraged to streamline processing earlier in the pipeline. Users can also utilize Filebeat for multiline processing at the data edge. These updates are part of Logstash's evolution, with further insights to be shared at the Elastic{ON}^16 user conference.

The weekly update on Elasticsearch and Apache Lucene discusses recent changes, improvements, and ongoing developments within the Elastic ecosystem. Key updates include enhancements in Elasticsearch such as re-enabling closures in the Groovy scripting plugin, improving translog recovery speed, and introducing the new Painless scripting language. The update also mentions the deprecation of term-level fuzzy queries and highlights changes to settings infrastructure, along with enhancements in deep pagination and task management frameworks. Apache Lucene is preparing for a 5.5.0 release, which will introduce backported features and debug the release process, while also transitioning from subversion to git. The update highlights improvements in geo queries, test coverage, and performance gains in geo point queries, alongside addressing test failures and refining various aspects of its API and build processes. The blog encourages readers to stay tuned for more updates and learning resources related to the Elastic ecosystem.

Elastic is hosting a series of events and meetups worldwide, focusing on the Elastic Stack and its various components such as Elasticsearch, Kibana, and Logstash. Notable events include the OOP Konferenz 2016 in Munich, where Alexander Reelsen will discuss the Elastic Stack, and linux.conf.au in Australia, featuring a tutorial on open data by Joshua Rich and Mark Walkom. There are also numerous meetups across Europe, North America, Africa, and Australia, covering topics like Elasticsearch installation, geolocation, and time-series data management. These gatherings provide introductions to Elasticsearch and its applications, as well as discussions on advanced features like visualizations using Timelion and the Graph API. Elastic encourages community engagement by offering support and promotional items for those interested in hosting or speaking at related events.