January 2021 Summaries

Elasticsearch 7.11 introduces significant optimizations to its date_histogram aggregation, a key feature used in Kibana's Discover and Logs Monitoring UI, by addressing performance issues related to daylight savings time transitions. Initially, the java.util.time APIs caused inefficiencies due to object allocations, prompting the development of a custom daylight savings time conversion code that leverages precomputed transitions and binary search, which significantly speeds up the process. Further enhancements involved transforming date_histogram aggregations into range and filters aggregations, offering up to an eightfold performance boost by using the search index instead of doc values. The new "filter by filter" execution mode, which optimizes filters aggregations, reduces maintenance burdens and opens possibilities for future aggregation optimizations. These improvements not only enhance performance but also provide a basis for further experimentation and benchmarking, with nightly tests conducted using JMH and Rally, and the Elastic team encouraging contributions and offering job opportunities for those interested in full-time work on these advancements.

Shell International leverages cutting-edge technology, such as Elasticsearch, to maintain its competitive edge in the global energy sector, which encompasses both renewable and non-renewable projects. At ElasticON 2020, Johan Krebbers, Shell's general manager of Digital Emerging Technologies, highlighted the importance of Elasticsearch's flexibility, which allows Shell to comply with various data retention regulations and supports a range of data types crucial for the company's operations. Shell utilizes Elasticsearch for predictive maintenance, employing machine learning to anticipate equipment failures, thus minimizing downtime and optimizing operational efficiency. Additionally, machine vision solutions help detect leaks and emissions through automated monitoring, while natural language processing enables swift analysis of extensive subsurface data to enhance exploration and development. These technologies collectively assist Shell in increasing uptime, reducing costs, and advancing its mission to harness new energy sources.

Elastic Security enhances threat investigations and collaboration by allowing users to export and import Timelines and Timeline templates between Kibana Spaces or instances. The Timeline feature serves as a dynamic workspace for investigations, enabling users to drag and drop queries and collect data from multiple indices to analyze complex threats. It auto-saves progress for team review and supports the creation of templates that filter out noisy alerts, ensuring a unified perspective on potential threats. To share a Timeline, users can export it as an ndjson file, which contains the necessary information to recreate the Timeline elsewhere. Importing a Timeline requires editing the savedObjectId in the ndjson file to prevent conflicts with existing Timelines, while templates allow for updates through file imports by adjusting the templateTimelineVersion. Elastic Security documentation offers further guidance, and new users can explore its features via Elasticsearch Service on Elastic Cloud.

Elastic App Search provides a scalable solution for creating personalized search experiences across applications and websites by leveraging user search history. By tagging each analytics event with a unique user ID, developers can build a search client and an external controller that tracks user search terms and clicked documents to generate personalized search results. This involves querying the App Search analytics API for a user's top queries over a specified time range and using these to suggest relevant documents, potentially excluding previously clicked items for novelty. The system is flexible, allowing segmentation by various criteria such as geographic region, and can incorporate additional user data for more tailored results. Developers are advised to handle the analytics API with care to maintain security and can experiment with the feature using a free trial of App Search on Elastic Cloud.

Kibana, a part of the Elastic Stack, allows users to create detailed maps using custom boundaries through reverse geocoding, which can enhance decision-making by transforming IP addresses into geographic coordinates. By using the GeoIP processor, Kibana can map user locations to varying levels of precision, such as city or neighborhood, which is more accurate than country-level mapping. The process involves creating, indexing, and integrating geospatial data and custom boundaries, such as Combined Statistical Areas (CSA) from the Census Bureau, into Elasticsearch. Users can enrich documents with CSA identifiers using a geo-match enrichment policy and a targeted pipeline. This enriched data can then be visualized in Kibana's Maps app, allowing for the creation of maps that highlight user activity hotspots in specific metro areas, which can be useful for marketing campaigns or stakeholder presentations. Overall, the ability to customize map boundaries in Kibana expands the scope of geographic insights available to users, encouraging them to explore new ways of mapping and analyzing data.

Elastic Community's January 2021 spotlight highlights the contributions of user group organizers in Latin America, showcasing their passion and dedication to the Elastic community. João Neto from Goiânia became an organizer to share his enthusiasm for technology and help others, motivated by his own journey with Elastic starting from a challenging project. Felipe Queiroz from São Paulo leveraged his interest in Elastic during the pandemic to create high-quality virtual workshops, reaching thousands and inspiring him to establish a virtual community group in Brazil. Adam Brandizzi, based in Brasília, turned to organizing as a way to maintain connections and share knowledge after transitioning to remote work, applying his experience with Elastic at Liferay. Victor Villas Bôas Chaves from São Paulo became an organizer to deepen his understanding of Elastic's capabilities, emphasizing the importance of community learning and knowledge sharing. These organizers, each with their unique backgrounds and stories, embody the spirit of collaboration and community that Elastic aims to foster, encouraging others to engage and contribute regardless of their technical expertise.

Elastic announced changes to the licensing of Elasticsearch and Kibana, moving from the Apache 2.0 to a dual license under the Elastic License and Server Side Public License (SSPL). The change primarily affects those who sell Elasticsearch and Kibana as a service, like Amazon's Elasticsearch Service, but not on-premises or Elastic Cloud customers. While the Elastic License restricts the direct sale of their products as a service and the modification of source code for paid features, the SSPL, although copyleft and not OSI approved, allows some freedoms typical of open-source licenses. Elastic is considering adopting a model similar to the Business Source License (BSL) for future versions, which balances business needs with open-source principles by allowing production use under certain conditions and transitioning to an open-source license after a set period. The company is seeking feedback on this potential licensing approach ahead of their next release.

Elastic's decision to change its licensing for Elasticsearch and Kibana stems from ongoing disputes with Amazon regarding trademark infringement and the provision of Elasticsearch as a service without collaboration. Despite efforts to resolve these issues through legal avenues and negotiations, Elastic's leadership felt compelled to revise the license to prevent further misuse and confusion within the open-source community. This move was prompted by Amazon's actions, which Elastic claims have misled users and fragmented the community, including the launch of Amazon Elasticsearch Service and the Open Distro for Elasticsearch fork. The licensing change aims to protect Elastic’s intellectual property while maintaining its commitment to open-source values, and it is not expected to impact existing users or customers.

Elastic announced a change in the licensing of its Elasticsearch and Kibana products, moving from an Apache 2.0 license to a dual license model under the Server Side Public License (SSPL) and Elastic License. This shift aims to provide users with free and open access to the software while protecting Elastic's investment and ensuring contributions from cloud service providers that offer these products as a service. The change, set to take effect before the 7.11 release, mirrors similar steps taken by other open source companies and comes in response to evolving market dynamics, particularly regarding SaaS models. Elastic emphasizes that the change will not impact most of its user community, as the default distribution remains free and open, and it continues to engage openly with its community. The SSPL requires that if the software is offered as a service, any modifications and management layer source code must also be publicly released under the same license. Elastic's commitment to transparency and open development remains steadfast as it adapts to protect its software and community.

Version 7.10.2 of the Elastic Stack has been released, bringing a series of fixes and minor enhancements to its components, including Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Workplace Search, App Search, Elastic Observability, and APM. Users are encouraged to upgrade to this latest version to take advantage of the improvements. Additionally, updates have been made to Elastic Cloud, Elastic Cloud Enterprise (ECE) version 2.7.1, and Elastic Cloud on Kubernetes (ECK) version 1.3.1, ensuring continued optimization and functionality across these platforms.

The Elastic Community Conference (ElasticCC) is a free, global technical conference designed for developers, practitioners, customers, and partners within the Elastic community. Despite being early in the year, the conference has already received 100 presentation submissions, with a call for more submissions extended until January 22, 2021. Scheduled to take place virtually on February 26-27, ElasticCC will feature a wide range of sessions covering diverse topics such as small business security, open-source collaborations, and unique use cases of the Elastic Stack, including healthcare systems and personal applications like managing diabetes. The event will also highlight contributions from Elastic employees and offer sessions in multiple languages to reflect the community's global nature. Attendees can look forward to engaging sessions and opportunities to win stickers and swag by participating in challenges. The conference emphasizes community engagement and offers a platform for sharing knowledge and innovative uses of Elastic technologies.

Elastic has introduced a series of free 15-minute Quick Start guides designed to help users quickly learn how to utilize Elastic solutions for enterprise search, observability, and security. Each guide consists of short videos focusing on specific topics, such as logging, and addresses why each topic is important, how to ingest data, and how to interpret it. These guides are available on demand and can be explored using an Elastic Cloud trial. The Quick Starts include tutorials on Elastic Workplace Search, Elastic Security, and User Experience Monitoring, each providing step-by-step instructions on using Elastic's features to optimize search experiences, detect threats, and monitor web apps. Elastic continues to offer various learning resources, such as webinars and hands-on training, to maximize the use of their solutions.

Creating a scalable and user-friendly web crawler for Elastic Enterprise Search involves addressing numerous challenges, including handling misbehaving sites, duplicate content, and noncompliance with web standards. The development team has leveraged their extensive experience since the first iteration of their web crawler for Swiftype Site Search, which now processes over a billion web pages monthly, to build an advanced content ingestion mechanism. Key issues tackled include deduplicating content, identifying content uniquely even when URLs are unreliable, and managing unpredictable crawl lifecycles due to various site-specific issues. The new crawler incorporates sophisticated URL and content hashing techniques, defensive mechanisms for parsing diverse content types, and built-in heuristics to handle complex scenarios efficiently. Observability is enhanced by integrating Elasticsearch, which logs every crawler action for detailed analysis using Kibana, ensuring transparency and allowing users to understand the crawler's decision-making process. Future updates and features are anticipated as the crawler continues to evolve, with users encouraged to explore its capabilities through Elastic's free trials and resources.

Elastic Cloud offers a managed service for deploying and utilizing the Elastic Stack, which includes Elasticsearch, Kibana, Beats, and Logstash, to provide powerful solutions for enterprise search, observability, and security. Users can choose to run Elastic Cloud on public clouds like Google Cloud, AWS, or Microsoft Azure, or opt for hybrid, private, or on-premises deployments, managing them with built-in automation and orchestration tools. The platform provides preconfigured solutions that can be customized to adjust capacity and performance based on user needs, and it supports a wide range of data ingestion options to enable robust monitoring and visualization capabilities through Kibana. Elastic Cloud also offers a free trial and easy sign-up process to help users get started quickly, with resources available for learning and community support. The service is tailored to accommodate different use cases, including government-regulated environments, with specialized offerings such as the FedRAMP-authorized cloud for secure data handling.

Audi Business Innovation (ABI) leverages Elasticsearch Service to streamline software development for brands like Audi, Volkswagen, and Porsche, significantly enhancing their capabilities in autonomous driving, electric vehicles, and in-car digital experiences. With the shift from a single on-premises Elasticsearch cluster to the scalable and flexible Elasticsearch Service available on AWS, Google Cloud, and Microsoft Azure, ABI can focus on developing software rather than managing infrastructure. This transition allows them to efficiently store and analyze 29 terabytes of log files, quickly addressing software issues to maintain a seamless customer experience, including the ability for customers to purchase in-car digital features post-purchase via mobile phones, car dashboards, or the web. Elasticsearch Service's low operational costs, high performance, and automation allow ABI to provide each application with its own Elasticsearch cluster and Kibana instances, fostering innovation while staying true to their motto of moving people and shaping the future.

The Elastic Contributor Program encourages community members to share knowledge and contribute to Elastic's open-source projects, with rewards available for various types of contributions, including code. To contribute code, contributors must ensure their submissions improve Elastic technologies and align with project guidelines, which can be found on GitHub. It's crucial to select a project that matches the contributor's skill set and language proficiency, as well as to understand the licensing requirements to avoid conflicts. Before submitting a pull request (PR), it's advisable to discuss the proposed changes with project maintainers to ensure relevance and avoid rejection due to redundancy or conflicts with existing plans. Contributions should focus on impactful areas like end-user code, reusable extensions, or security fixes. Once approved, contributors can submit their PR to the Elastic Contributor Program and are encouraged to share their work on social media to recognize their efforts.

The text discusses the process of migrating from a self-managed Elasticsearch setup to the managed Elasticsearch Service on Elastic Cloud, specifically on AWS. It highlights the advantages of using Elasticsearch Service, such as handling infrastructure management, scaling, and upgrades, allowing users to focus on other tasks. The migration involves taking a snapshot of the existing cluster and restoring it on the Elastic Cloud. This requires configuring cloud storage, setting up a snapshot policy, and provisioning a new cluster on Elasticsearch Service. The guide provides detailed steps, including configuring AWS S3 for storage, setting permissions, and ensuring successful restoration of the snapshot. The text emphasizes the ease and benefits of transitioning to Elastic Cloud, encouraging users to consider the move for streamlined management and efficiency.