May 2019 Summaries

The Elastic{ON} Tour is a series of global one-day events organized by Elastic that offer expert advice from engineers, leadership talks, and AMAs, providing a valuable opportunity for the Elastic community to engage with the company's technology and each other. The events are meticulously planned and executed, with the team arriving a day early to set up and ensure smooth operations, often working late into the night. At a recent stop in Seattle, key figures like VP of Worldwide Marketing Jeff Yoshimura and Senior VP of Engineering Kevin Kluge presented updates and roadmaps, while Elastic engineers engaged with attendees, answering questions and discussing use cases. The event also involves significant logistical efforts, including on-the-spot problem solving and a focus on delivering a high-quality experience for attendees. The Elastic videographers play a crucial role, capturing and editing the event content to provide prompt access to the recorded presentations. The Tour not only highlights the latest developments in the Elastic Stack but also reinforces community ties by offering a platform for interaction and feedback.

The text celebrates Bike Month at Elastic, a company with roots in Amsterdam, a city renowned for its cycling culture. Employees from various global locations share their personal experiences and the role bikes play in their lives, from commuting and family outings to competitive racing and leisure activities. The narratives highlight the benefits of cycling, such as reducing carbon footprints, promoting health, and providing a sense of freedom and exploration. The stories also reflect a shared enthusiasm for biking, extending beyond Amsterdam to influence lifestyle choices and advocacy for better cycling infrastructure worldwide. The text underscores the bicycle's versatility and its impact on personal and community well-being, demonstrating its significance as more than just a mode of transportation.

CDL, a leading UK tech firm, utilizes innovative data solutions to process vast amounts of consumer data for high-volume retail operations, helping industries combat fraud and understand consumer habits. By leveraging the Elastic Stack, CDL has developed Hummingbird, a high-speed data analytics solution that provides real-time insights into consumer behavior and fraud prevention. The firm employed Canvas to visualize key performance indicators (KPIs) in a more business-friendly manner, improving the monitoring of metrics relevant to management audiences. Canvas dashboards were created to enhance visibility into service KPIs and transaction journeys within Hummingbird, showcasing metrics such as response times and the number of requests processed. This visualization capability has been expanded to tackle insurance fraud by identifying real-time changes in consumer data entries. CDL's consistent application of branding elements across dashboards ensures a recognizable and cohesive customer experience, aligning with the company's strategy to utilize technologies like machine learning and AI to meet digital consumer expectations.

MuleSoft has become a preferred choice for integrating systems and automating business processes due to its lightweight and user-friendly nature, avoiding the complexities often associated with other integration technologies. However, managing the observability of MuleSoft components, especially in distributed environments, can be challenging when relying on varied and bespoke monitoring tools. This article introduces the use of Elastic Stack, particularly Elastic APM, to enhance the monitoring of MuleSoft environments. By integrating the Elastic APM Java Agent with Mule runtime, users can collect runtime transaction performance and metric data, providing a comprehensive view of Mule and non-Mule deployments on a single dashboard. The post details how to set up the Elastic Stack for MuleSoft, including instrumenting Mule projects with Maven and configuring them for both local and CloudHub deployments. The use of Elastic’s machine learning capabilities and advanced visualizations in Kibana further enhances the monitoring experience, offering insights into performance metrics and system health.

Elastic has released the fifth iteration of its self-managed, on-premises Elastic App Search beta, introducing significant updates such as support for self-signed Elasticsearch SSL certificates and compatibility with Elasticsearch 7.1.0. Additionally, an optional telemetry feature has been added to allow users to send usage statistics to Elastic, with the assurance that data will remain confidential within the company. Users of previous beta versions are advised to start afresh with beta 5 as the product nears general availability. Elastic encourages users to test the updated version and provide feedback, emphasizing the importance of user input in refining the product.

Elastic has announced a global partnership with Tencent Cloud to offer their Elasticsearch Service on the Tencent Cloud platform, which will enhance access to both free and paid Elastic Stack features for users in China and worldwide. This collaboration aims to provide seamless integration and localized support, leveraging Tencent's extensive reach and technology prowess. The partnership ensures that the Tencent Cloud Elasticsearch Service will include critical security features and proprietary enhancements such as Kibana localization, alerting, monitoring, and more. This move aligns with Elastic's strategy to collaborate with leading global cloud providers, reinforcing its commitment to the Chinese market by expanding its local presence, partner ecosystem, and dedicated support infrastructure. The service, consistent with Elastic's own offerings and those of other platform partners, enables users to utilize Elastic's full suite of solutions across different cloud environments, supporting diverse applications like enterprise search, logging, and security.

CreatorIQ, a SaaS marketing platform, leverages Elasticsearch Service on Elastic Cloud to enhance its search capabilities for identifying suitable social media influencers for brands, moving away from their initial SQL database setup which was inadequate for the growing influencer marketing industry. Initially struggling with limited resources, CreatorIQ found Elastic Cloud's comprehensive support, machine learning, and security features superior to AWS Elasticsearch Service, helping them manage the complexities of social media API changes and reindexing challenges. The transition enabled CreatorIQ to focus on their core competencies of social media analysis and workflow management, while also ensuring reliable service delivery through Elastic's support, which proved crucial during critical times, such as a cluster failure before a major client demo. This move not only improved their operational efficiency but also led them to upgrade to a Gold subscription with Elastic, providing better resource allocation and response times essential for meeting their service level agreements.

In a significant update to the Elastic Stack, Elasticsearch announced that its core security features are now available for free in the default distribution starting with versions 6.8.0 and 7.1.0. Previously part of a paid Gold subscription, these features include TLS for encrypted communications, user management, and role-based access control, which also secures Kibana Spaces, enhancing security for cluster operations, particularly in Kubernetes environments. This change aligns with the release of Elastic Cloud on Kubernetes (ECK), the official Kubernetes Operator designed to streamline Elasticsearch deployment and operations within Kubernetes. While advanced security features remain part of paid tiers, the integration of basic security into the free tier aims to ensure seamless, secure setups for users across all environments. Existing users are encouraged to upgrade to the latest versions to benefit from these enhancements, with support materials available to facilitate the process.

Elastic Cloud on Kubernetes (ECK) is a new orchestration product announced by Elastic, designed to facilitate the deployment, management, and operation of Elasticsearch clusters within Kubernetes environments using the Kubernetes Operator pattern. ECK offers a comprehensive suite of features that include managing and monitoring multiple clusters, streamlining upgrades, scaling capacities, and ensuring secure deployments by default. It extends beyond a typical Kubernetes Operator by integrating the full Elastic Stack experience, including free features like Canvas, Maps, and Uptime, and supports advanced cluster topologies, such as hot-warm-cold configurations. Built on Elastic’s extensive operational expertise, ECK promises seamless orchestration and operational efficiency for Elasticsearch on Kubernetes, with an open and transparent forever-free tier, alongside an Enterprise subscription offering enhanced capabilities. Initially supporting Google Kubernetes Engine and vanilla Kubernetes, Elastic aims to broaden its compatibility with other Kubernetes flavors in future releases, ensuring a consistent and robust experience for users across cloud and on-premises environments.

Starting with Elastic Stack versions 6.8 and 7.1, security features such as TLS encrypted communication and role-based access control (RBAC) are available for free, enabling users to secure their Elasticsearch clusters effectively. The blog post provides a practical guide on setting up these security features by creating and securing a two-node Elasticsearch cluster on a local machine. It outlines steps for configuring TLS between nodes, enabling security in Kibana, and setting up RBAC to ensure users have access only to authorized data. The process involves downloading and setting up Elasticsearch and Kibana, creating certificates for secure communication, configuring roles and users in Kibana, and ensuring proper access control. The post emphasizes that while the guide covers essential security setup, there are many additional features and capabilities within the Elastic Stack security framework to explore, encouraging users to refer to comprehensive documentation for further learning.

The adoption of the Elastic Common Schema (ECS) in the Elastic Stack 7.0 marked a significant shift in data ingestion, enhancing observability by standardizing fields across logs, metrics, and traces. This schema simplifies the correlation of events, making it easier to track and analyze system performance and issues. By unifying fields such as host names and IP addresses, ECS facilitates seamless data queries across different sources, thereby improving the functionality of Elastic's Infrastructure, Logs, APM, and Uptime UIs. The implementation of ECS reduces field discrepancies, allowing for better data visualization and understanding within these UIs. Additionally, ECS supports cross-cluster search, enabling users to access and analyze data from multiple Elasticsearch clusters, which is particularly beneficial for global operations. As ECS evolves, it promises to further streamline data management, encouraging users to map their own data to the schema for consistent analysis and interpretation.

Elasticsearch introduced a new mapping type called "alias" that enables users to define alternative names for fields within an index, enhancing search capabilities without requiring reindexing. This feature is particularly beneficial for renaming fields in time-based indices, as it allows users to implement new field names while maintaining access to older data. Field aliases also facilitate migrating indices to a common schema, such as ECS, by allowing searches across indices with different field names. In practical applications, such as monitoring telemetry data from the International Space Station, field aliases help standardize unit specifications in field names without disrupting historical data analysis. While aliases can be employed in search requests, queries, aggregations, and other search functionalities, they cannot be used when indexing documents, as they only apply to the search layer rather than the document source. This makes field aliases a lightweight alternative to data re-writing when the primary need is for search functionality rather than indexing.

King Abdullah University of Science and Technology, a graduate research institution in Saudi Arabia, has significantly enhanced its infrastructure monitoring by adopting the Elastic Stack, a decision influenced by its licensing model and community spirit. Previously, the university faced challenges with disparate and inflexible monitoring tools, which hindered efficient data visualization and alerting. With the Elastic Stack, managed by Stanislav Flachs and Gianluca Castellani, the university has achieved improved system visibility and operational efficiency, storing heterogeneous data, integrating with communication tools, and providing key performance indicators to management. The scalable and intuitive platform supports multiple use cases, including capacity planning and resource management. Future plans involve implementing security features, employing machine learning for anomaly detection, and further exploring the platform's functionalities to enhance intrusion detection and system resilience.

Elasticsearch audit logs can be effectively indexed and analyzed using Filebeat, especially after the deprecation of the index output type in version 6.7.0 and its complete removal in version 7.0.0. Filebeat, a separate process from Elasticsearch, takes on the task of indexing audit logs, relieving the Elasticsearch nodes from this load and allowing them to focus solely on storing events. By configuring Filebeat alongside Elasticsearch, administrators can manage audit logs more efficiently and take advantage of features such as event filtering and sending logs to external systems like Logstash or Kafka for further processing. This setup not only prevents data loss during high load scenarios but also enhances analysis capabilities in Kibana by allowing the correlation of various types of logs. Additionally, Filebeat's configuration flexibility and ability to handle multiple Elasticsearch clusters provide robust options for managing logs across distributed environments.

The new Elasticsearch JavaScript client has been officially released after a month of development and feedback integration, offering production-ready features and improvements. This client supports JavaScript and TypeScript, and users can easily install and run it using npm along with Elasticsearch. Key features include enhanced observability, hostname sniffing, improved type definitions, and support for custom HTTP agents. The library size has decreased, and the documentation has been significantly improved. The new observability features allow for better tracking of events by providing request IDs and client names, which are particularly useful in complex environments with multiple child clients. The client also offers comprehensive TypeScript support, with type definitions for its entire API and improved developer experience through the use of generics. Users are encouraged to try the client locally or on the Elasticsearch Service and provide feedback through various platforms.

Brittany Joiner reflects on her experiences working in a distributed environment at Elastic, highlighting the personal and organizational benefits of this flexible work model. She appreciates the freedom to travel and work from various locations, which enhances her productivity and allows her to align her work schedule with her natural energy levels. Joiner also notes the cost savings associated with working from home, like reduced commuting expenses and the convenience of having meals at home. From an organizational perspective, distributed work fosters a results-oriented culture at Elastic, where communication is prioritized through digital platforms like Zoom and Slack, enhancing collaboration among global teams. She emphasizes that while distributed work offers numerous advantages, it requires intentional communication and an understanding of diverse working styles to be successful. Elastic's culture supports this by encouraging face-to-face interactions during offsites and annual gatherings, and by promoting a respectful and non-malicious approach to digital communication.

Elasticsearch terms aggregations, which create buckets for unique values in a field, can experience slow performance due to factors like cluster misconfiguration and high cardinality, where fields contain numerous unique values. High cardinality can significantly impact performance, as it involves extensive computation of global ordinals, a data structure that numbers unique terms for efficiency. Techniques to mitigate these performance issues include using time-based indices to limit the need for recalculating global ordinals, enabling eager global ordinals to precompute data structures during refreshes, and opting not to build global ordinals at all by executing terms aggregations directly on raw terms, although this may increase memory consumption and reduce efficiency. Adjusting the refresh interval and tuning cluster settings can also enhance performance, and tools like Elasticsearch logs and the hot_threads API can help diagnose issues related to global ordinals.

Elasticsearch is a versatile application that can experience slow query performance due to a variety of factors, including shard management, thread pool rejections, and resource contention. To address these issues, strategies such as reducing shard count, adopting a hot/warm architecture, and optimizing index and search performance are suggested. The document emphasizes the importance of capacity planning, using recommended hardware, and configuring settings like index.refresh_interval and filesystem cache allocation. Additionally, it highlights the role of adaptive replica selection (ARS) and circuit-breaking strategies in handling occasional and consistent slow queries. The use of slowlogs and audit logs can aid in identifying and addressing slow or expensive queries. Overall, the document provides a comprehensive approach to diagnosing and resolving performance bottlenecks in Elasticsearch queries, while encouraging users to leverage community resources for further assistance and optimization insights.

Elasticsearch is introducing a new feature that incorporates proximity, both geographical and temporal, into its result ranking system to enhance relevance scoring. This feature, known as the distance feature query, allows for a more nuanced ranking by integrating proximity with traditional criteria such as word frequency, overcoming previous technical challenges related to normalization and performance. By using the Saturation normalization function, the proximity score is normalized, making it easier to combine with other scores, while performance improvements allow the ranking to consider a broader set of records. The main advantage is the ability to better rank records by considering both proximity and other relevance criteria without significant performance degradation, addressing the need for more relevant results in scenarios where location or time is crucial. This development emerged from community feedback and contributions, leading to new algorithms and APIs that facilitate more efficient top-hit retrievals, ultimately showcasing Elasticsearch's commitment to evolving and enhancing its search capabilities.

Elastic Enterprise Search, which was later renamed to Elastic Workplace Search, is a consumer-grade enterprise search solution designed to unify and simplify access to scattered data across various cloud-based tools that organizations use for productivity. The product allows for fast, scalable, and relevant searches from a single search bar, capitalizing on Elasticsearch's proven relevance and scalability. It offers integration with popular applications like Google Drive, GitHub, Salesforce, and Dropbox, alongside a Custom Connector Framework for syncing data from any source. Elastic Enterprise Search provides features such as advanced keyword detection, privacy groups, and tunable search relevance, enabling users to efficiently find information tailored to their team's needs, all while maintaining security through granular access controls. The introductory beta version aimed to showcase its capabilities with community feedback, and while not initially fit for production usage, it promised seamless data indexing, blazing speeds, and expert support from Elastic.

Elasticsearch 6.7 and 7.0 introduced cross-cluster replication (CCR), a feature designed to replicate indices across clusters for purposes such as disaster recovery and geographically distributed deployments. To ensure the performance, resilience, and stability of CCR, Elasticsearch expanded its benchmarking tool, Rally, to communicate with multiple clusters and collect metrics from new CCR-related API endpoints. Benchmarking involved creating a topology with a leader cluster in Europe and a follower cluster in North America, ensuring a network latency of at least 100ms. The tests included various load scenarios to ensure followers could keep up, optimized remote recovery processes, and evaluated the impact of network compression on recovery times. Stability was tested over 10 days, simulating real-world conditions with index lifecycle management and random restarts of clusters. An accessible recipe using Docker was also developed to allow users to benchmark CCR in their local environments.

Indexing documents into Elasticsearch using the NEST Elasticsearch .NET client can be performed through various methods, ranging from indexing single documents to handling multiple documents with advanced techniques. For single documents, the IndexDocument and IndexDocumentAsync methods provide straightforward approaches, while the fluent or object initializer syntax offers more control. When dealing with multiple documents, using the Bulk API is more efficient than iterating through individual index calls, and the BulkAllObservable helper further optimizes this process by managing retries and batching mechanics. The blog post also discusses creating ingest pipelines to manipulate and enrich data before indexing, utilizing features such as the ingest-geoip plugin to convert IP addresses into human-readable locations. For large datasets, considerations like partitioning collections into smaller batches and adjusting timeouts are necessary to avoid errors during bulk operations. Overall, the NEST client offers multiple configurations and helper functions to facilitate efficient document indexing in Elasticsearch.