January 2016 Summaries

Konrad Beiske's article discusses the process and benefits of using Elastic Cloud, previously known as Found, for cloning Elasticsearch clusters in the cloud. By leveraging the Snapshot and Restore API, users can efficiently duplicate clusters without impacting the performance of the source cluster, providing flexibility for various use cases such as ad hoc analytics, testing upgrades, enabling developers, testing mapping changes, and integration testing. The article details the steps required to clone a cluster, including preparing the target cluster, selecting a snapshot, and issuing a restore, while highlighting the importance of compatibility and non-conflicting indexes. The data transfer occurs from backups in S3, ensuring no performance implications on the source cluster during the restore process. Elastic Cloud's cloning capabilities extend its automated backup service, allowing users to manage their clusters with ease while benefiting from the isolation and scalability provided by separate clusters.

In the blog post, Andrew Cholakian introduces a new Logstash codec plugin, logstash-codec-nmap, which integrates Nmap scan results into Elasticsearch, allowing for visualization with Kibana. Nmap, a versatile network tool, is used for various networking tasks such as pinging hosts, issuing traceroutes, and checking open ports, among others. The post explains how to use Nmap to generate XML output, which can then be imported into Logstash via the Nmap codec. Cholakian provides a detailed setup for a Logstash server to receive and process Nmap data, which can be visualized in Kibana to create network monitoring dashboards. Additionally, he explores setting up a small network monitor using Elasticsearch and Kibana to analyze network activity and visualize outbound network routes. The post emphasizes the flexibility of using Logstash with Nmap and suggests potential future developments in the codec's capabilities.

"Where in the World is Elastic?" highlights a series of Elastic-related events and meetups occurring globally during the week of January 25, 2016. These events span multiple continents, including Asia, Europe, North America, and Australia, and offer participants opportunities to engage in workshops, discussions, and networking sessions focusing on various Elastic products like Elasticsearch, Kibana, and Beats. Highlights include hands-on workshops in Oslo, a Q&A session in London with notable figures such as Mark Harwood and Rashid Khan, and a hack night in Dallas. The meetups aim to foster community engagement and knowledge sharing among users and developers of the Elastic Stack, with additional support offered for those interested in hosting or presenting at future events.

Mapp's Root Cause Analysis (RCA) team leverages the Elastic Stack, consisting of Elasticsearch, Logstash, and Kibana, to enhance their logfile analysis capabilities, transitioning from traditional Unix command-line tools to more powerful solutions for managing large volumes of data. By implementing the Elastic Stack, the team can conduct real-time full-text searches, manage diverse log formats, and create interactive Kibana dashboards that facilitate data sharing and analysis across teams. This setup not only improves the speed and accuracy of log analysis but also fosters a proactive approach to incident management and allows cross-functional teams to better understand and address the consequences of system issues. The Elastic Stack's adaptability supports various input types and integrates with additional data sources, promoting an agile and lean operational environment that aligns with modern practices like Agile, DevOps, and Microservices. As the team continues to refine their processes, they are exploring further enhancements with tools like Shield and Watcher to address multitenancy and alert management within their infrastructure.

Elastic has initiated a community engagement activity called #ElasticSnap, inviting participants to share photos of themselves wearing Elastic-branded gear in various global locations. This tradition, rooted in the company's distributed nature and inspired by its CTO's preference for black t-shirts, aims to visually connect Elastic's widespread community. As Elastic{ON} 16, a major gathering for Elasticsearch users, approaches, the campaign seeks to showcase the community's spirit by collecting high-resolution images of people using Elasticsearch or working on Elastic Stack projects. Participants can submit their photos via email or Twitter using the hashtag #ElasticSnap, with a chance to win an Elastic cluster t-shirt, encouraging creativity and public sharing.

In a detailed exploration of Elasticsearch queries, the author discusses the performance dynamics of various query types, emphasizing the speed and efficiency of term queries, particularly when compared to alternatives like phrase, prefix, and fuzzy queries. The text highlights that term queries are notably fast, especially with the use of the `terminate_after` parameter, which limits the number of hits processed, enhancing performance for unselective queries. Despite the convenience of the `query_string`, the author advises against its use for user-generated queries due to its complexity. The author also notes that phrase queries can be efficient when terms are uncommon, but their performance deteriorates with common terms. Additionally, the text outlines the impact of query configurations, such as fuzziness and prefix length, on performance, advising preferences for AND over OR queries when possible. The discussion concludes with practical advice on optimizing query performance, suggesting the use of direct term and match queries, considering `terminate_after` for specific use cases, and exploring other performance-enhancing techniques like the shingle token filter and rescore wrapping for phrase queries.

Ansible has emerged as a user-friendly alternative to Puppet for deploying Elasticsearch clusters, leveraging its push architecture and Python-based automation to streamline the process across heterogeneous hardware. With the release of Elasticsearch 2.0.0, a new Ansible role simplifies the installation and configuration of Elasticsearch nodes, supporting tasks from prerequisite installation to service management. This role is particularly beneficial for organizations without a history of using Puppet, as it requires only SSH access and Python 2 on the target host, eliminating the need for an agent or external dependencies. The Ansible role's structure facilitates the assignment of Elasticsearch instances to hosts, and the configuration process is adaptable through a map serialized to YAML, ensuring ease of updates. An example using an Ubuntu-based Docker image demonstrates the role's capabilities, outlining steps for setting up a playbook and deploying Elasticsearch nodes. This role is part of a broader effort to provide Ansible roles for other Elastic products like Beats and Logstash, aiming to deploy the entire Elastic stack with ease and encouraging community contributions and feedback.

Deepdetect is an open-source deep-learning server designed to integrate machine learning into existing technology stacks with minimal effort, particularly by connecting to data back-ends like Elasticsearch. Originating from applications developed for both large enterprises and startups, Deepdetect supports the Caffe library and uses output templates to seamlessly index tagged images into Elasticsearch, enabling text-based image searches even without captions. This approach is scalable, allowing for batch processing and parallel predictions across multiple servers. The text highlights the growing role of machine learning as a ubiquitous technology in diverse fields, emphasizing the ease with which complex tasks such as image classification can now be automated through deep learning models, capable of high-level data abstraction using neural networks and GPU parallelization. The article also notes the potential of machine learning as a developer commodity, akin to Elasticsearch in storage and search applications, and discusses the evolution of deep learning frameworks and their applications across various domains, including cybersecurity and intelligent software solutions.

"Where in the World is Elastic?" provides a schedule of upcoming Elastic events and meetups happening globally during the week of January 18, 2016. Highlights include David Pilato's half-day training on Elasticsearch and Kibana at SnowCamp in Grenoble, France, and Elastic's presence at the SCaLE 14x conference in Pasadena, California. In Europe, notable meetups include a Search Meetup in Munich hosted by PAYBACK, featuring talks on Elasticsearch use cases and time-series databases, and an Apache Kafka Meetup in London. North America features several Elastic User Group Meetups, including an event at Ticketmaster's office in Los Angeles and a discussion on Elasticsearch use cases in the WordPress community in Denver. Asia and Africa are represented by meetups in Bangalore, India, and Cape Town, South Africa, respectively, with topics covering the Elastic Stack's applications. The Elastic Team encourages community involvement by offering support and merchandise to those interested in hosting or speaking at future Elastic-related events.

In Jay Greenberg's article from January 18, 2016, the evolution of network infrastructure is discussed, highlighting the challenges and solutions related to system logging and monitoring in modern, cloud-based environments. The text emphasizes the importance of self-monitoring systems and introduces the Beats platform, an open-source data shipper that integrates with Elasticsearch to streamline performance metrics and business reporting. Greenberg points out the inefficiencies of traditional polling methods, advocating for a more unified approach that leverages automation and configuration management technologies. The article details how Nagioscheckbeat can be configured to monitor services like Redis, and how Elastic’s Watcher plugin can alert users to critical system statuses, using examples of alert configurations and heartbeat monitoring. The integration of Elasticsearch with existing monitoring systems is presented as a secure and scalable solution, and the use of cloud services like Found and PagerDuty is recommended to simplify architecture and operations. The overall message is that by adopting modern tools and practices, businesses can achieve a stable and intelligent network infrastructure.

The Logstash Lines update, dated January 12, 2016, outlines the preparations for the Logstash 2.2 release, highlighting key enhancements and fixes. The team focused on optimizing the default number of pipeline workers to fully utilize all CPU cores, which improved performance in common scenarios like Apache log processing. New features include warnings for overly large batch sizes, updated event documentation, and an API to declare output plugin thread safety. Manageability improvements introduced a bootstrap config file to enhance command line options, and plugin updates included refactoring for Beats Input to fix thread synchronization issues, new settings for File Input to manage file descriptors, and enhancements to ES and Kafka outputs, including SSL support for Kafka. Additionally, JDBC input was improved to allow query run state to be saved using various column types.

Elastic's "Where in the World is Elastic?" initiative highlights a series of meetups occurring globally, focusing on various Elastic Stack components like Elasticsearch, Logstash, and Kibana. Notable events include talks on Logstash at the Manitoba UNIX User Group in Canada and deploying the ELK Stack at the Boston AWS Meetup Group. In Europe, Christoph Wurm will discuss predictive analytics on Elasticsearch at the Not Only SQL Tel Aviv Meetup, and new features of Elasticsearch 2.0 at the Elastic Tel Aviv Meetup. The Elastic Team encourages community involvement and offers support for those interested in hosting or speaking at meetups related to their technologies.

Elasticsearch for Apache Hadoop 2.2 RC1 has been released, marking a significant step towards its general availability, with numerous bug fixes and enhancements introduced. This release candidate offers support for Spark 1.6, optimizing performance by skipping redundant filters and improving push-down translation, particularly with IN filters. Enhancements to Spark SQL include better handling of multi-valued and array fields, and improvements in connector configuration, addressing issues with discarded command line properties. Updates to the YARN module involve upgrading to Elasticsearch 2.1.x and allowing JVM system properties to be passed in children containers. The repository HDFS plugin is undergoing significant changes to align with Elasticsearch's security improvements, with a migration to Elasticsearch core planned for version 2.3. Network improvements address varied topologies, enhancing the handling of Elasticsearch clusters, and reducing configuration requirements for users. The developers invite feedback through various platforms, looking forward to further developments in 2016.

AbilityNet, a charity that assists disabled individuals in utilizing digital technology, significantly benefits from the support of Elastic, a rapidly growing technology provider known for its search, logging, and analytics software. Elastic's funding plays a crucial role in enhancing AbilityNet's web-based resources, which are the primary means for people to access the charity's expert knowledge, including factsheets, blogs, webinars, and the interactive guide "My Computer My Way." This collaboration was highlighted at the Elastic{ON} Tour in London, where AbilityNet was featured as a charity partner, allowing them to receive financial support and network with industry professionals. The partnership enables AbilityNet to continue upgrading its resources, thereby empowering more individuals to make small but impactful changes to accessibility features on various digital devices.

Elasticsearch 2.2 introduces significant improvements to its geospatial capabilities, particularly with the enhancement of geo_point fields through a new GeoPointField type built on the internal inverted index structure. This update leverages a quad-tree raster graphics approach to encode latitude and longitude values, improving efficiency and reducing index size by minimizing the terms dictionary. The two-phase query approach has been optimized to maximize the coverage area and minimize brute force checks, resulting in better performance for complex geo queries. Additionally, the update streamlines mapping parameters by removing or simplifying certain options, such as coerce and doc_values, which are now inherently managed by the new structure. Despite these advancements, ongoing efforts aim to further enhance Elasticsearch’s geospatial capabilities with specialized data structures for multi-dimensional spatial data in future releases.

After receiving mixed feedback from their inaugural Elastic{ON} conference, the organizers decided to enhance the upcoming 2016 event in San Francisco by expanding the popular AMA booth and addressing complaints about uncomfortable seating. They tested various high-quality chairs at their Mountain View office, eliminating some due to issues like lack of support and discomfort. Ultimately, chairs 4, 5, and 6 remained contenders, each offering full back support and varying fabric options. Despite the focus on choosing the perfect chair, the organizers acknowledged the importance of regular breaks and encouraged attendees to stretch during the conference. They expressed excitement about the upcoming event and hoped to see participants there.

In January 2016, Found, a company specializing in hosted Elasticsearch services, announced a major milestone by integrating the full Elastic stack into its offerings, highlighted by the addition of the Marvel monitoring plugin. This achievement follows Elastic's acquisition of Found in March 2015, which was driven by a shared culture and vision, enabling Found to expand its customer base and service capabilities. The integration of all Elastic products into Found's services marks a significant advancement for the company, which started as a small Norwegian startup in 2012 and quickly gained traction in the cloud hosting space. As Found continues to grow, the company is planning future developments, including a fully packaged Elastic stack for on-premise and private cloud environments, aimed at meeting the increasing demand for Elasticsearch and supporting businesses without necessitating expanded operations teams.

In this blog post, Pius Fung demonstrates how to integrate Zendesk with the Elastic Stack to enhance data discovery and visualization capabilities by utilizing a custom Logstash plugin. Despite having no prior experience in Ruby, Fung successfully developed the plugin by leveraging the Zendesk Ruby API Client, allowing for the extraction of various Zendesk objects such as organizations, users, tickets, and comments. Once the data is ingested into Elasticsearch, tools like Kibana can be used to search and visualize the information, while the Watcher plugin enables monitoring and alerting on product quality trends. To ensure data security, the Shield plugin provides authentication, access control, and encryption features. The blog also highlights the convenience of deploying the Elastic Stack on Found, a managed service that simplifies the setup process and ensures data protection through automatic backups, allowing users to focus on building dashboards and other tasks without operational concerns.

In the development of Kibana, the team uses various automation tools to streamline their workflow, particularly when dealing with Elasticsearch. They employ "makelogs" to efficiently index fake HTTP traffic logs into Elasticsearch, allowing them to quickly visualize data and conduct initial tests on new clusters. However, the scope of "makelogs" is limited to Kibana development. A more versatile tool they use is "esvm" (Elasticsearch Version Manager), which assists in managing multiple Elasticsearch versions and clusters with unique configurations. "esvm" can download specific versions of Elasticsearch, manage cluster settings via JSON configuration files, and facilitate the installation of plugins. This automation reduces the complexity and time involved in setting up necessary infrastructure, enabling the team to focus more on developing Kibana itself. Additionally, the team is working on a Kibana plugin generator to further ease plugin development, hinting at future improvements for users.

Upgrading from Elasticsearch 1.x to 2.x involves several critical changes that require careful planning to ensure a smooth transition. The process mandates a full cluster restart and involves familiarizing oneself with the breaking changes detailed in the new release documentation. Key steps include setting the network host correctly in the elasticsearch.yml file, utilizing the migration plugin to identify necessary changes, and leveraging the snapshot and restore feature to test the upgrade in a non-production environment. It is essential to upgrade associated tools like Logstash and Kibana to compatible versions, with specific attention to plugin compatibility and configuration adjustments. In the event of issues, users are encouraged to seek support through community forums or customer support channels. For those seeking to minimize downtime, an alternative approach is to build a new cluster on the updated version, ensuring all data and functionalities are transferred before decommissioning the old system. Known issues, such as plugin compatibility and field mapping conflicts, must be addressed to prevent upgrade failures.