April 2016 Summaries

The blog post delves into how Kibana, a data visualization tool, manages object persistence by storing saved searches, visualizations, dashboards, and other objects within Elasticsearch. These objects are stored as documents in a special index called .kibana, which is created when Kibana is first installed and started. The post explains how different types of documents are created in this index, such as config for storing version information, index patterns for user-defined data structures, and the specific fields each document type contains. It also covers the processes involved in saving searches, visualizations, and dashboards, including the fields stored in each type of document. The post highlights that these internal data structures may evolve, with future versions potentially offering REST APIs for safer manipulation. Additionally, it notes the current limitations and future-proofing measures in place, such as reserved schema version fields and non-implemented features like popularity tracking. The post aims to assist both in debugging and in deploying Kibana in a consistent and automated way, and it acknowledges the contributions of Rashid Khan and Spencer Alger in designing the document structures.

Structured logging involves directly writing JSON objects from applications to avoid the need for parsing logs into JSON via regular expressions, thus facilitating efficient indexing into Elasticsearch. This approach is exemplified in Python using the structlog library, which simplifies log statements and encourages comprehensive data inclusion. Although structured logs are less human-readable, they are well-suited for Elasticsearch's search and aggregation capabilities. Filebeat, an open-source log shipper, can send structured logs to Logstash and Elasticsearch, offering "at-least-once" delivery guarantees and features like native JSON decoding, basic filtering, and metadata addition. It simplifies log processing by automatically adding timestamps and metadata such as host names, allowing applications to focus on generating essential log data without worrying about environmental metadata.

Elasticsearch 2.3.2, based on Lucene 5.5.0, has been released as a bug fix update, addressing significant issues from previous versions and is available on the Elastic Cloud platform. Key improvements include fixes for CORS-related problems affecting browser-based applications, adjustments to ensure proper shard allocation decisions, and corrections for the extended_stats aggregation handling of empty buckets. The release also enhances various accompanying tools: Marvel now includes a timepicker control for viewing historical data, Shield resolves date math expressions for index names before authorization, and Watcher sets SMTP connection timeouts to prevent watches from stalling. Additional fixes ensure smoother functionality across the suite, such as handling of HTTP headers with dots and Hipchat action fallbacks. Users are encouraged to download and test the update, with feedback welcomed via Twitter or the Elastic forum.

Elastic, the company behind popular open-source projects such as Elasticsearch, Logstash, and Kibana, is actively engaging its community by hosting a series of events and meetups worldwide. In the coming weeks, Elastic will be present at various gatherings across North America, Europe, Asia, and Australia, including DevOps Days in Austin and Puppet Camp in Sydney. Community-driven meetups are scheduled in cities like Portland, Dallas, New York, Berlin, and Beijing, where enthusiasts and users can connect and share insights. The Elastic team encourages individuals interested in hosting or speaking at meetups to reach out for support and promotional materials, fostering a collaborative environment for discussions on topics related to Elasticsearch and its ecosystem.

Running Elasticsearch within Docker containers requires careful consideration of plugin management and data persistence, particularly for stateful services. This involves using Docker's basic bind mounts to store Elasticsearch data outside the container, ensuring that changes persist beyond the life of ephemeral containers. Plugins can be managed by extending Docker images using a Dockerfile, which facilitates repeatable and trackable installations. More complex plugins that require additional files or configurations benefit from keeping images generic to enable re-use and maintain tighter control over sensitive data. For instance, the article discusses using Shield for access control and SSL/TLS for secure communication, highlighting the importance of managing configuration and persistence through volume mounts. The overall approach emphasizes reliability, repeatability, and security, urging users to tailor setups for their specific environments while maintaining persistent data separately from container images to ensure data control and container ephemerality. Testing is crucial to ensure that Elasticsearch operates as expected within Docker, especially regarding network communication and plugin functionality.

Rally is a benchmarking tool developed by the Elasticsearch team to evaluate Elasticsearch's performance, allowing developers to reproduce performance numbers and create their own benchmarks without dealing with complex details. Originating from Python scripts used for nightly benchmarks, Rally aims to prevent gradual performance degradation and improve the performance of Elasticsearch by enabling developers to run benchmarks independently during development. It supports features like telemetry devices for detailed analysis, allowing inspection of Java flight recorder and JIT compiler behavior, and facilitates visualization of metrics data with Kibana. Rally currently supports single-machine benchmarks and is working towards allowing multi-machine benchmarks, improving measurement correctness, and separating benchmark definitions from the tool itself. It simplifies the process of running benchmarks, with the potential for more flexibility in benchmark scenarios and the elimination of current limitations such as coordinated omission in latency measurements.

Elastic is actively engaging with global tech communities by hosting and participating in a variety of events and meetups across continents, including in Asia, Europe, North America, Australia, and South America. Notable upcoming events include QCon in Beijing, Devoxx France, and AWS Summit in Chicago, alongside numerous Elastic User Group Meetups in cities such as Chicago, San Francisco, Paris, and Melbourne. These gatherings provide opportunities for networking and sharing knowledge about Elastic's technologies, such as Beats, Elasticsearch, Logstash, and Kibana. The company encourages community involvement by offering support and promotional materials to those interested in hosting meetups or giving talks.

The blog discusses network considerations for using Docker with an Elasticsearch cluster, specifically focusing on different Docker networking setups: none, host, bridge, and overlay. The none network disables networking, making it unsuitable for Elasticsearch, while the host network, though offering high performance, poses security risks. The bridge network is the default setup where containers have virtual Ethernet interfaces connected to the docker0 bridge, allowing internal container communication and external connectivity through IP forwarding and iptables. For Elasticsearch, specific configurations are needed to ensure external accessibility and node discovery, such as setting the network.publish_host and configuring Zen Discovery. Docker containers are configured for IPv4 by default, but can be switched to IPv6 for inter-host communication. Overlay networks, recommended for multi-host setups, require a key-value store for node discovery, with Docker supporting Consul, etcd, and ZooKeeper for this purpose.

Elastic{ON} 16, an annual user conference for Elastic Stack enthusiasts, offered a dynamic experience filled with engaging presentations and community interactions. Attendees had the opportunity to meet other users, interact with the development team, and explore various aspects of the Elastic Stack through numerous talks and community chats. The event featured a range of presentations across multiple stages, with topics like Logstash, Elasticsearch's geospatial data structures, and Kibana plugin development among the highlights. The conference also showcased new features such as Graph capabilities, improved text scoring with BM25, and Ingest Node for document enrichment. Some talks, such as those on cluster sizing and securing Elasticsearch, achieved perfect ratings from attendees, reflecting their popularity and relevance. The event's success is attributed to the vibrant community and insightful content, with plans already underway for the next gathering, Elastic{ON} 17, in San Francisco.

Elastic, a platform known for its search and data analytics capabilities, is organizing a series of events and meetups across the globe to engage with its community. In North America, events include Percona Live in Santa Clara and an AWS Summit in Chicago, along with several Elastic User Group meetups in cities like Los Angeles, Chicago, and San Francisco. Europe hosts a variety of gatherings, such as Code.talks in Berlin and PyCon in Ukraine, while Asia features QCon in Beijing. Elastic also has meetups scheduled in Australia with its Melbourne User Group, and in South America with PanamaJS. The Elastic Team encourages those interested in hosting related events or giving talks on their technologies, such as Beats, Elasticsearch, Logstash, or Kibana, to reach out for support and promotional materials.

Elasticsearch for Apache Hadoop versions 2.3.0 and 2.2.1 have been released, featuring compatibility improvements and bug fixes that encourage users to upgrade promptly. These releases include important updates such as HDFS repository compatibility with Elasticsearch 2.3.0, marking the last release cycle with the HDFS plugin in ES-Hadoop 2.3. Additionally, network transfer for fixed routing is optimized to target specific shards, indexing of Spark RDDs is improved to prevent unnecessary loading, and the detection algorithm for overlapping shards has been enhanced to use less memory, allowing it to handle more indices. ES-Hadoop 2.2.1 is the final maintenance release of the 2.2.x line, incorporating backported fixes for users on a conservative upgrade path, with strong recommendations for upgrading to ES-Hadoop 2.3 even for those using ES 1.x. Feedback is encouraged through various platforms including GitHub, Twitter, forums, and IRC.

Creating a Dockerfile for Elasticsearch offers a customizable approach to deploying applications in isolated environments. The process begins with a base image, such as Ubuntu, and involves installing necessary components like Oracle JDK 8 and Elasticsearch itself. By configuring files such as `elasticsearch.yml` and `logging.yml`, users can tailor the setup to their needs, though it is crucial to handle logging and data persistence appropriately. Docker volumes are recommended for persisting data across container restarts, while memory constraints and heap size settings ensure efficient resource management. This setup provides a foundational image for Elasticsearch, with potential enhancements like networking and plugin installations to be explored in future discussions.

USAA, a financial institution serving the U.S. military community, successfully transitioned from traditional security information and event management (SIEM) solutions to the Elastic Stack, significantly enhancing their data management and cyber threat prevention capabilities. This shift, led by Nelly Cyrus from the company's Cyber Threat Operations Center, resulted in cost savings and improved productivity for security analysts, as they could now proactively hunt for malicious activity using advanced logging and monitoring tools. The transition to a multi-cluster setup using Elasticsearch allowed USAA to manage vast amounts of data more efficiently, reducing the time for data snapshots and enabling quick responses to potential threats. The Elastic Stack's open-source nature and robust support from Elastic's engineers were pivotal in advocating for its adoption within the company, demonstrating tangible benefits such as increased analyst productivity and scalability in handling billions of security events daily.

Logstash versions 2.3.1 and 2.2.4 have been released, providing significant compatibility and security updates, and users are encouraged to upgrade immediately. The Logstash 2.3.1 release addresses several issues, including a regression in regex handling due to an upgraded JRuby version, which introduced a thread safety bug. Environment variable support was added in 2.3.0 but is now disabled by default in 2.3.1 due to compatibility problems with existing configurations, though it can be enabled via a command-line flag. The new Java implementation of the Event class was reverted to the previous Ruby version due to compatibility issues, but the Java version is expected to return in future releases. Additionally, Logstash 2.3.1 fixes a bug where plaintext passwords were printed in log files under certain conditions and resolves a broken configuration test flag from the 2.3.0 release. Users are invited to provide feedback on the release through various channels.

The Elastic Stack 5.0 has been released in its alpha 1 version, allowing users to test new features and improvements before its general release, as part of the Elastic Pioneer Program. This initiative encourages users to identify and report bugs during the pre-release phase, as the software is tested in diverse and unforeseen environments. Participants who report legitimate bugs will receive recognition and a special Elastic gift package, with the possibility of earning a free ticket to Elastic{ON} 17. The program aims to ensure the software is as refined as possible before its broader distribution, emphasizing that these alpha releases are intended solely for testing and should not be used in production environments.

During Elastic's annual conference in San Francisco, attendees were asked to describe Elasticsearch in three words, capturing a variety of insights about the tool. Many participants highlighted its speed and scalability, reflecting the core strengths of Elasticsearch. The event, which also featured a more informal session at the Elastic{ON} party, provided a platform for users to express their experiences and opinions about the software. For a deeper understanding of the diverse perspectives, viewers were encouraged to watch the accompanying video.

The article, "Where in the World is Elastic?" highlights various upcoming Elastic events and meetups scheduled over the next two weeks across multiple continents, including North America, Europe, Asia, and South America. Key events include the Lone Star PHP conference in Dallas, Texas, and several Elastic user group meetups in cities such as Milwaukee, Los Angeles, and Amsterdam. The piece encourages readers to engage with the Elastic community by attending these gatherings, and it offers support for those interested in hosting their own meetups or giving talks on Elastic-related topics such as Beats, Elasticsearch, Logstash, or Kibana. The article concludes with an invitation to stay updated on future Elastic happenings.

Kibana 5.0.0-alpha1 has been released, introducing a new design that focuses more on data presentation by minimizing unnecessary navigation elements, in contrast to its predecessor, Kibana 4. This alpha version, which requires Elasticsearch 5.0.0-alpha1, includes enhancements such as first-class applications, allowing plugins to be integrated more prominently into the main navigation alongside core features like discover, visualize, and dashboard. Furthermore, Kibana 5 introduces a simplified plugin installation process, enabling users to install multiple plugins as a single pack, making it easier to enhance functionality with third-party and native plugins like Timelion and X-Pack. Although this release is not yet production-ready and includes known issues, the developers are actively working on further improvements and encourage user feedback through various channels.

Over a year after attending a tutorial on the Elastic Stack at Strata + Hadoop World, Salesforce's Adam Torman and Abhishek Sreenivasa presented at the Elastic{ON} 16 conference on their integration of Salesforce Event Monitoring with the Elastic Stack. Inspired by their initial exposure to the Elastic Stack, Abhishek developed a plug-in with intern Mohammed Islam, aimed at visualizing event and log data. Their conference presentation, titled "Users: WE KNOW THEM," highlighted the plug-in's design and construction, with notable connections made, including with Kurt Hurtado, their original tutorial instructor. The conference emphasized networking and collaboration, allowing attendees to engage in insightful discussions about event management and data visualization, while also enjoying the event's unique and engaging atmosphere. Adam, with over a decade of experience at Salesforce, and Abhishek, a software developer focused on platform monitoring, showcased their expertise in enhancing Salesforce's capabilities through innovative solutions in event monitoring and security.

Elasticsearch 5.0.0-alpha1, based on Lucene 6-SNAPSHOT, introduces several new features and enhancements designed for testing and feedback as a precursor to the official 5.0.0 release. Notably, it includes the introduction of dimensional points in Lucene 6, which optimizes disk space and indexing speed for numeric, date, and geospatial fields. The release also features the Ingest Node, which simplifies data processing by integrating popular Logstash filters directly into Elasticsearch. A new scripting language called Painless is introduced, offering faster and safer scripting capabilities. Other enhancements include instant aggregations for improved caching, the introduction of text and keyword field types for better data handling, a revamped completion suggester, and strict settings validation to prevent configuration errors. The update also emphasizes safety in production with improvements in resilience, such as cluster state updates and index management, ensuring data safety and reliability.

Elastic announced the release of the Elastic Stack 5.0.0 alpha 1, marking a significant shift towards integrating and evolving its suite of products as a cohesive unit. The new version introduces X-Pack, a bundled extension that combines features from previously separate products like Marvel, Shield, and Watcher into a single solution, enhancing security, alerting, monitoring, graph, and reporting capabilities across the entire stack. The update also brings enhancements to individual components such as Elasticsearch, which now features Ingest Node for data processing and Lucene 6 for improved search performance, and Kibana, which sports a new design and application framework. Logstash gains event statistics and hot threads API, while Beats adds flexibility with custom fields and JSON support, alongside a new Kafka output. Though still in alpha and not yet available on Elastic Cloud, the release aims to simplify development and foster community feedback, with further improvements expected in the coming months.

The first Seoul Elasticsearch meetup of 2016 took place on March 31 at Naver D2 Startup Factory in Gangnam, attracting around 130 attendees—nearly three times the usual number. The event featured several presentations, including a review of Elastic{on} 2016 and a demonstration of the new Graph feature in Elasticsearch 2.3.0 by Jongmin Kim, an auto-tagging system using Elasticsearch by Junyi Song from Melon, and a discussion on building search infrastructure in e-commerce by Ho-wook Jung from Memebox. Additionally, Geonbok Lee from Microsoft Korea demonstrated creating an Elasticsearch cluster using Microsoft Azure Marketplace, humorously engaging the audience with a nod to open-source by declaring "We love Linux." The meetup's popularity was evident as it quickly reached full capacity and topped the OnOffMix popularity rankings shortly after being announced. The event also marked the introduction of Elastic's new stickers in Korea and featured greetings from several key Elastic representatives, including Robert Lau, Chin Lian Heng, Matias Cascallares, and Hawon Jung.

Managing the heap size in Elasticsearch, which runs on the Java Virtual Machine (JVM), involves a delicate balance due to its impact on allocation speed, garbage collection frequency, and application performance. A heap that is too small can lead to frequent pauses and out-of-memory errors, while a heap that is too large can cause long pauses due to full-heap garbage collections, affecting the system's responsiveness and stability. The ideal heap size should stay below the 32 GB threshold to benefit from compressed ordinary object pointers (oops), which optimize memory usage and performance. However, it's recommended to set the heap size as low as possible to meet application requirements, as this allows more physical memory for the filesystem cache, enhancing performance. The G1 Garbage Collector, introduced in JDK 7u4 and set to be the default in JDK 9, helps manage larger heaps with predictable pause times by focusing on regions with the most garbage. Properly configuring the heap size and understanding the intricacies of memory management, including the advantages of zero-based compressed oops, can prevent performance issues and improve the efficiency of Elasticsearch systems.

Elasticsearch 2.3.1, based on Lucene 5.5.0, has been released as a bug fix update, addressing a thread deadlock issue in Shield that could prevent nodes from joining the cluster. This release is available on Elastic Cloud, the Elasticsearch-as-a-service platform, and users of Elasticsearch 2.3.0 with Shield are advised to upgrade. Feedback is encouraged via Twitter or the forum, and any issues can be reported on the GitHub issues page.