June 2015 Summaries

Kibana 4.1 introduced field formatters, a feature that allows users to display data in their preferred format, regardless of how it is stored in Elasticsearch. This functionality enables users to customize the presentation of data, such as displaying date values in a specific format or formatting raw number values like bytes. Field formatters are versatile, allowing users to create links, insert images, and even represent HTTP status codes with creative visuals like HTTP status cats. To format fields, users can navigate through the Kibana interface by selecting the appropriate index pattern, choosing the desired field, and then specifying the format. This feature enhances the usability and visual appeal of data presented in Kibana.

Elasticsearch provides insights into its operations through logging, which can help identify errors, performance issues, or undesirable cluster behaviors. Main logs are stored in a default INFO level file, while slowlogs reveal the speed of queries and other operations. Log levels can be dynamically adjusted without restarting nodes, allowing for detailed monitoring of specific Java packages or sections within Elasticsearch. This flexibility extends to slowlogs, allowing index-level logging adjustments to focus on specific indices without affecting others. Groovy scripts can include logging to debug and verify script functionality, and with the advent of Elasticsearch 2.0, a deprecation log was introduced to inform users of deprecated settings and features.

"Where in the World is Elastic?" is a blog post by Livia Froelicher that outlines a series of events and meetups related to Elastic and its technologies, such as Elasticsearch and Kibana, happening worldwide from June 30 to July 4, 2015. The post highlights key events in Europe, North America, and the Asia Pacific, including JDEV2015 in Bordeaux, PolyConf 2015 in Poznan, and various Elastic meetups across several cities. It also offers a call to action for individuals interested in hosting a meetup or giving talks about Elastic's technologies, offering support and promotional materials in return. The post indicates a slight slowdown in events with the arrival of summer but assures readers of more Elastic activities in the coming weeks.

Elasticsearch has introduced significant changes to its type and field mapping system to address longstanding issues of ambiguity and inconsistency that have caused exceptions, incorrect query results, and even data corruption. With the release of Elasticsearch v2.0, stricter rules are enforced for new indices, requiring that fields with the same name within an index have identical mappings, thereby preventing data corruption and ensuring consistent query outcomes. The changes necessitate that users resolve any conflicting mappings by either using separate indices or renaming fields, among other strategies. Additionally, the update simplifies meta-fields and analyzer settings, removing certain configurable options to streamline processes and prevent mapping conflicts. While these changes aim to enhance reliability and performance, they require users to adjust their current practices, especially when upgrading to v2.0, and may involve reindexing data or using tools like the Elasticsearch Migration Plugin to identify and resolve mapping conflicts.

This Week in Elastic, published on June 24, 2015, provides a comprehensive overview of updates, tips, and events related to the ELK Stack, which comprises Elasticsearch, Logstash, and Kibana. The article highlights significant releases such as Elasticsearch for Apache Hadoop 2.1 GA with native support for Spark and Storm, the Logstash 1.5 update featuring Apache Kafka integration, and Watcher 1.0.0-RC1 introducing dynamic variables via scripting. The piece also includes various tutorials and success stories, like DHL's use of Shield for securing logistics data, Vox Media's optimization of video analytics, and TrueCar's use of Elasticsearch for personalized car shopping experiences. Additionally, the article lists numerous global meetups and training sessions, underscoring the widespread use and community engagement around the ELK Stack, inviting developers and users to participate in events, and offering insights into best practices for using these technologies.

Elasticsearch for Apache Hadoop 2.1 is a general availability release that enhances Hadoop workflows with a real-time search and analytics engine, built on feedback from four successful beta versions. This release introduces significant improvements, including native integration with Apache Spark and Storm, enabling efficient indexing and analysis of data directly from Spark and exposing Elasticsearch as a Spout or Bolt in Storm topologies. It also includes security enhancements, such as SSL/TLS encryption and PKI authentication, and introduces Elasticsearch-on-YARN for deploying clusters within a YARN environment. The update supports Spark versions 1.0 through 1.4 and offers enhanced functionality like JSON document returns, document metadata retrieval, and selective field inclusion/exclusion for data transformation. The release maintains backward compatibility and is certified with major platforms like CDH, HDP, and MapR, providing users with a single JAR file for straightforward upgrades and no external dependencies.

On June 24, 2015, Jay Modi announced the release of Shield 1.3 and Shield 1.2.2, with Shield 1.2.2 addressing bug fixes and Shield 1.3 introducing new features, including the addition of the Public Key Infrastructure (PKI) realm for authentication without storing user credentials. Shield 1.3 also incorporates a new index-based output for auditing, allowing audit events to be indexed for search and analysis using Elasticsearch. Notable changes include the removal of the sha2 and apr1 hashing algorithms for the cache.hash_algo setting, requiring users to switch to supported alternatives or the default ssha256, and the users file now only supporting bcrypt password hashes, which aligns with the esusers tool. Users are directed to the Shield documentation for detailed upgrade instructions and are encouraged to provide feedback in the Shield forums.

Apache Kafka integration with Logstash 1.5 introduces both input and output plugins, enhancing Logstash's ability to process and handle messages flowing through Kafka's distributed system. Kafka's scalable persistence allows it to act as a message broker, buffering messages between Logstash agents, which is useful in complex, CPU-intensive Logstash processing pipelines. The integration enables data to be written to a Kafka topic using the stdin input plugin and read from Kafka using the elasticsearch output plugin, with consumer groups allowing for scalable read throughput. Kafka's design supports a healthy buffer of events, facilitating the addition of more brokers to handle growing data volumes. Current plugins use the older Kafka producer, with plans to release an updated version in Logstash 1.6, bringing significant API changes that will not be backwards compatible. Future posts will explore serialization with Kafka, including the use of Apache Avro.

Elastic is actively participating in numerous events and meetups across four continents, engaging with the global community on topics related to their technology offerings. In North America, Elastic is present at several events, including the Red Hat Summit in Boston, where they offer sessions and interactions at their booth, and the GEOINT Symposium in Washington DC. Europe sees Elastic's involvement in events like DevOps Days in Amsterdam, with talks and hands-on training sessions in Krakow. Additionally, Elastic is organizing and participating in various meetups across North America, South America, Europe, and Asia, aiming to foster community discussions around their tools such as Elasticsearch, Logstash, and Kibana. The Elastic Team encourages community engagement and offers support for those interested in hosting meetups or giving talks about their technologies.

In the weekly roundup "This Week in Elastic," Leslie Hawthorn presents the latest developments and updates in the ELK Stack ecosystem, highlighting the release of Elasticsearch for Apache Hadoop 2.1 RC1 with enhancements for Spark SQL and security. The post covers the removal of rivers from Elasticsearch, urging users to transition to alternatives like Logstash. It also features community contributions, such as a video tutorial by Dimitris Ganosis on using the ELK Stack and a guest blog by Search Technologies on their Aspire framework for unstructured data. The roundup promotes upcoming global meetups and events, including training sessions and conferences across various countries, to engage and educate users and developers about Elasticsearch, Logstash, and Kibana. Additionally, the post encourages feedback on the new JDBC input plugin and emphasizes the vibrant community involvement in expanding the ELK Stack's reach and capabilities.

LeadBoxer and Opentracker, co-founded by Cralan Deutsch and Wart Fransen, leverage Elasticsearch to power real-time analytics for B2B sales and web tracking. Initially using MySQL, they transitioned to Cassandra for scalability but lacked real-time search capabilities. To address this, they incorporated Elasticsearch, which allowed flexible, real-time searches and informed the creation of LeadBoxer, a tool for capturing and qualifying leads based on online activity. By manipulating Elasticsearch's scoring algorithms, they developed a customizable lead scoring system, enhancing sales teams' ability to target potential leads effectively. The integration of Elasticsearch enabled the delivery of qualified leads with adjustable scoring preferences, facilitating personalized sales strategies. This approach has positioned the companies to potentially transform B2B and B2C sales processes by combining external data with user behavior insights, aiming to present clients with tailored sales opportunities.

Aspire for Elasticsearch, developed by Search Technologies, is a content access and processing framework designed to handle unstructured data for Elasticsearch implementations. It enables users to acquire, clean, normalize, and enrich content from diverse repositories, supporting document-level security through access control lists from source repositories. Aspire is particularly valuable for organizations needing to manage unstructured content, as seen in a management consulting firm that used Aspire to index web content and large databases. It comes with pre-packaged connectors for various content stores like SharePoint, Box.com, and Salesforce, and its modular framework allows for easy integration and licensing of additional connectors. Aspire also addresses security concerns by complementing Elasticsearch's Shield, which provides foundational security features such as encryption and role-based access controls. By tackling the inconsistencies typical of unstructured content, Aspire enhances the effectiveness of search and analytics applications, exemplified by a recruiting company that used it to normalize date formats and case usage in resumes. Additionally, Aspire's integration with Hadoop facilitates advanced features like vector creation for document matching. The framework is available as a 45-day trial, with options for full licensing, and includes features such as CIFS, RDB snapshot, and Heritrix connectors, as well as support for Elasticsearch for Hadoop.

Elastic is actively engaging with the global community through a series of events and meetups, highlighting their presence at significant conferences such as Monitorama in Portland, Big Data Analytics in London, and GOTO Amsterdam, where they offer insights into their technologies like Elasticsearch and Kibana. In London, they feature a Kibana 4 demo and a guest speaker from the Met Office, while in Amsterdam, they provide a dedicated track on Elasticsearch with various expert talks and Dutch waffles at their booth. Numerous meetups are scheduled across North America and Europe, emphasizing their commitment to fostering discussions and knowledge-sharing within the tech community. Elastic also encourages community involvement by supporting those interested in hosting meetups or giving talks about Elasticsearch, Logstash, or Kibana.

Watcher 1.0.0-Beta2 has been released, featuring bug fixes and enhancements to the features initially introduced in Beta1, thanks in part to user feedback that improved its functionality, usability, and documentation. The development team actively incorporated user suggestions, demonstrating a commitment to community-driven improvement. The installation process for Beta2 remains straightforward, requiring only a simple modification to the existing Beta1 registration key. This release encourages new users to register and experience the improved version of Watcher.

This weekly update from Elastic highlights recent developments and events related to the ELK Stack, which includes Elasticsearch, Logstash, and Kibana. It covers the release of Elasticsearch version 1.6.0, featuring a new upgrade API, and the Logstash 1.4.3 bug fix release, recommending users to upgrade if they use certain plugins. Kibana also released version 4.1 with performance tweaks. The announcement includes various meetups, webinars, and conferences worldwide focusing on the ELK Stack, illustrating community engagement and educational opportunities. Additionally, the update mentions training sessions in multiple locations and encourages community members to participate and share their experiences at events. The post emphasizes the importance of community involvement and offers support for meetups and conferences, as well as highlighting job opportunities within the Elastic team.

Logstash 1.4.3 has been released as a bug fix update addressing critical security vulnerabilities in the 1.4 series, with an emphasis on upgrading for users of specific plugins such as Elasticsearch output with node protocol, Logstash Forwarder with Lumberjack input/output, and File Output. The release notably addresses vulnerabilities found in Elasticsearch 1.1.1, bundled with Logstash 1.4.2, which was susceptible to CVE-2014-3120, allowing command execution on the host OS via a node client's URL endpoint. The update packages Logstash with Elasticsearch 1.5.2, which disables script execution by default, and advises against using embedded Elasticsearch clusters in production. Additionally, the combination of Logstash Forwarder and Lumberjack was vulnerable to the POODLE attack in SSLv3, leading to the disabling of SSLv3 in favor of TLSv1.0. The release also fixes a File Output plugin vulnerability that allowed path traversal outside the Logstash directory, restricting the use of dynamic field references with absolute paths. An issue with socket connection release in the Elasticsearch output was also resolved, benefiting any plugins using the ruby-ftw HTTP client library.

Elasticsearch 1.6.0, based on Lucene 4.10.4, introduces several enhancements aimed at improving performance, security, and user experience. Key features include faster restarts with synced flushing, which minimizes data copying during node maintenance and accelerates recovery times. Shard allocation processes have been optimized to prevent blocking of pending tasks, enhancing cluster responsiveness. The release also includes JSON response body filtering to reduce network latency by allowing users to specify the exact data needed. Security has been tightened for shared file-system repositories, and an upgrade API has been introduced to facilitate the transition to future Elasticsearch versions by upgrading ancient indices. Kibana users benefit from more lenient highlighting, addressing issues with inappropriate fields and large terms. Additionally, Windows users now have access to the mlockall feature for improved node performance, and fine-grained script settings allow for more detailed control over script execution permissions.

The text highlights Elastic's participation in various events and meetups across North America and Europe in June 2015. Key events include the Hadoop Summit in San Jose from June 9-11, where visitors can meet local developers and learn about data exploration, and dotScale Paris on June 8, featuring swag and interactions with Elastic's local team. Additionally, David Pilato is scheduled to deliver talks on advanced search at Breizh Camp in Rennes and Riviera Dev in Sophia-Antipolis, while Alexander Reelsen will engage with attendees at Daho.am in Munich. The schedule also lists several Elastic-related meetups in cities like Seattle, Montreal, and Vienna, inviting community participation and offering support for those interested in hosting Elasticsearch, Logstash, or Kibana talks.

Elastic, the company behind the ELK Stack, has launched new discussion forums to enhance community interaction and continues to share updates and insights across its ecosystem, including Elasticsearch, Logstash, and Kibana. The recent blog posts and tutorials cover various applications and improvements, such as the Sunlight Foundation's use of Elasticsearch to manage US criminal justice data, scaling tips from Chartbeat, and a Mozilla Engineering team's optimization of Elasticsearch indexing time. Elastic's global presence is evident through numerous meetups and conferences worldwide, where community members can engage with Elastic's experts and learn about topics ranging from entity-centric indexing to performance tuning under heavy load. Upcoming training sessions are also planned in major cities, offering hands-on workshops and core Elasticsearch courses led by core developers. Elastic invites participation and support for community meetups and encourages sharing of knowledge and experiences, while also actively recruiting individuals passionate about their technology.