October 2018 Summaries

In modern deployments where application servers often have short lifespans or use unreliable hardware, Elastic APM Server, starting from version 6.4.0, can send data to more reliable systems like Logstash or Kafka, thereby offloading reliability requirements. This setup allows for data enrichment through Logstash plugins and involves configuring the APM Server to output events to Logstash or Kafka, which then forwards them to Elasticsearch. Each event type, such as transactions or spans, generates a daily index, and specific configurations are necessary to ensure the correct handling of APM data. Source maps, essential for mapping obfuscated code to original sources, require special consideration and should ideally be sent directly to Elasticsearch. Kafka can be introduced to buffer events before they reach Elasticsearch, with configurations allowing for topic-based organization of data. The integration of these systems enables more robust and flexible handling of application performance monitoring data, with the Elastic APM team encouraging feedback and contributions from users.

Elastic places a strong emphasis on family-inspired creativity, weaving personal connections and artistic expression into its corporate culture. Employees often draw inspiration from their families, with children and family members contributing to design projects, particularly for Elastic's events and promotions. The narrative highlights stories of employees whose children, like Kiran, Ryan, and Abraham, have created artwork that became part of Elastic's promotional materials or merchandise, demonstrating the company's commitment to integrating personal stories and creativity into its projects. This approach fosters a unique environment where creativity is nurtured and celebrated across all levels of the organization, emphasizing the importance of maintaining a connection to the uninhibited creativity of childhood. Elastic's culture encourages contributions from all employees and their families, reflecting a tradition of inclusion and artistic expression that enriches its community and brand identity.

CitiGroup, with a vast global presence, faces significant IT challenges due to its complex infrastructure and demands for greater efficiency. To address these issues, CitiGroup's IT teams implemented an integrated monitoring system using the Elastic Stack to centralize data, configure agents, and reduce costs. This system ingests vast amounts of data daily to monitor tool performance, offering a centralized data store and enabling data governance with role-based access control. Elasticsearch facilitates data storage, retrieval, and analysis, allowing for advanced search analytics and visualization. The system also supports container infrastructure monitoring, delivering comprehensive and real-time insights into application health through Kibana dashboards, which are used for audit findings, alert management, and various operational tasks. This setup empowers CitiGroup to monitor its IT infrastructure effectively, providing the necessary data to management for informed decision-making and operational improvements.

Elastic App Search has introduced support for multiple users, allowing up to 100 team members per account to collaborate on enhancing search experiences across various applications and platforms. This feature facilitates teamwork by enabling developers and non-developers alike to utilize the Elastic App Search dashboard, which offers tools such as analytics for tracking and optimizing search results based on user behavior. Users can refine search outputs by creating synonym sets, curating results, and performing relevance tuning, all within the dashboard. The platform's flexibility allows it to be integrated into diverse applications, from ecommerce sites to administrative dashboards, providing significant value regardless of the user's technical expertise. The new multi-user functionality, coupled with a 14-day free trial, seeks to reduce the complexity of implementing search solutions and encourages collaborative search optimization efforts.

Kibana's weekly update from October 2018 highlights various ongoing developments and improvements across its platform, focusing on enhancing security features, localization, alerting services, and cloud testing for Canvas. The team is close to implementing granular application privileges and has been working on translating visualization and dashboard elements while finalizing internationalization tools. Cloud testing has revealed some issues that are being promptly addressed, and the development of workpad templates is set to release ahead of schedule. Furthermore, a read-only mode has been introduced for demonstration purposes, and server-side execution has been improved for better performance management. Auto-complete features have been successfully integrated, along with security enhancements and bug fixes. The design team is advancing the Sass conversion, preparing for the K7 layout, and developing a new visualization builder, while operations and QA teams focus on patch releases and code optimization. Overall, the update reflects a productive period of technical advancements and cross-team collaboration within the Kibana project.

SAP Concur, a leading travel and expense management solution, transitioned from a SQL-based logging system to a more scalable and efficient solution using the Elastic Stack to manage the immense volume of logs generated by its extensive user base. Initially, Concur's logging system struggled with performance issues due to the high data ingestion rates, reaching limits that caused substantial delays. To address this, Concur adopted the Elastic Stack, which offered speed, scalability, and user-friendly visualization through Kibana, replacing various costly and disparate visualization tools previously used by different teams. This transition enabled Concur to handle increased data volumes effectively, with adoption rates and data ingestion soaring to 60,000 documents per second by 2017. They further enhanced their system with cross-cluster search, improving security to meet GDPR requirements, and integrating machine learning for operational analytics. Concur's LAMA team, consisting of six engineers and two managers, manages this sophisticated logging environment, facilitating end-to-end application ownership and DevOps practices across the organization.

In Elasticsearch 6.0, a feature called index sorting was introduced, allowing documents to be sorted by specified keys during indexing, which can enhance search performance and reduce disk space usage. By pre-sorting documents, Elasticsearch can bypass sorting during query time, leading to faster query responses, especially when sorting by the same keys. This method can also improve query performance by enabling Elasticsearch to skip non-matching document blocks efficiently. Additionally, when values are sorted and repeated, compression becomes more effective, resulting in significant disk space savings; the extent of these savings depends on the field cardinality and can vary greatly. However, enabling index sorting comes with a trade-off, as it may slow down indexing speed by up to 40-50%, making it unsuitable for high-volume indexing scenarios. It is most beneficial for use cases with lower indexing rates, where query speed is prioritized, and regular reindexing is feasible.

October 23, 2018, blog post discusses the integration of the Elastic Stack, Wazuh, and Suricata for enhanced security analytics, focusing on threat detection and incident response. The Elastic Stack enables efficient indexing and searching of security-related data, with Kibana dashboards allowing for interactive threat hunting, while its machine learning engine automates the analysis of complex datasets to identify potential intrusions. Wazuh, a host-based intrusion detection system (HIDS), and Suricata, a network threat detection engine, utilize signature-based threat detection to analyze patterns in files, logs, and network traffic. The integration of these tools within a lab environment involved deploying Wazuh agents on servers and a Suricata sensor for network traffic monitoring. This setup allowed for the unification of alerts, with Wazuh processing Suricata alerts and sending enriched security events to Elasticsearch, where machine learning jobs detected anomalies. An example of this system's efficacy was demonstrated when a machine learning job identified an abnormal IP address, leading to an automated Wazuh response that temporarily blocked the malicious source, showcasing the benefits of combining signature-based and anomaly-based detection techniques.

With the end of Google Search Appliance (GSA) approaching in 2019, Elastic Site Search Service emerges as a compelling alternative, offering seamless transition and enhanced functionality for site crawling and search integration. Previously known as Swiftype Site Search, this service provides a cloud-based crawler with intelligent indexing features, allowing users to implement robust and relevant search experiences on their websites with minimal effort. Elastic Site Search, built on Elasticsearch, offers scalability and performance, enabling organizations of any size to maintain high-speed search capabilities without downtime. Users benefit from real-time analytics and insights, customizable search results through features like result rankings and synonyms, and multi-language support, which enhances search precision and user satisfaction. Companies like Azusa Pacific University have successfully migrated to Elastic Site Search, noting improvements in both cost and user experience. Migration is facilitated through a straightforward process, supported by Elastic's dedicated Search Specialists, making it an optimal choice for organizations transitioning from GSA.

Efficient duplicate prevention in event-based data within Elasticsearch involves generating unique identifiers for documents before indexing to avoid duplication, which can lead to incorrect analyses and search errors. Two primary methods for creating these identifiers are the use of Universally Unique Identifiers (UUIDs) and hash-based identifiers, each with distinct advantages and potential drawbacks in terms of uniqueness and indexing performance. UUIDs, generated at the event's origin, offer a high level of uniqueness but may not be feasible in all systems, while hash-based identifiers depend on the event content and can be assigned later in the processing pipeline. Elasticsearch's internal identifier generation optimizes indexing performance, but when external identifiers are used, performance can be impacted due to required update checks. To balance performance with duplicate prevention, strategies such as timestamp-prefixing and utilizing the rollover and split index APIs can be employed, although these have implications for managing index sizes and maintaining the link between event timestamps and their indices. It is recommended to benchmark and tailor these approaches to specific use cases to ensure optimal performance and accuracy in event data management.

Perform Group, a sports media company, revamped its API strategy to address scalability and flexibility issues resulting from its previously fragmented architecture, which relied heavily on custom APIs tied to an Oracle RDBMS. This transformation led to the development of Perform Feeds, a unified API system leveraging Elasticsearch for enhanced search functionality, performance, and horizontal scalability. Implemented as Java web applications deployed to Tomcat servers, the new system decouples from the Oracle database, using RabbitMQ for change notification and Elasticsearch for data storage. Since its initial deployment in 2012, Perform Feeds has become integral to Perform Group's operations, supporting products like DAZN and websites such as goal.com and Sporting News, while also powering B2B services like Watch & Bet. The system has achieved high availability and rapid response times, handling millions of requests during peak periods, and is maintained by a distributed team across three locations. Additionally, Perform Group utilizes the Elastic Stack for log aggregation and is exploring further applications such as security analysis and machine learning for server load pattern detection.

Canvas, a presentation tool integrated into Kibana, allows users to create dynamic, pixel-perfect presentations and slide decks that pull live data directly from Elasticsearch, eliminating the need for manual updates. Released with Kibana version 6.5 and later, Canvas offers the flexibility to adjust data on the fly during presentations, enhancing their interactivity and relevance. It simplifies the process of creating presentations by providing tools and filters that enable users to manipulate data in real-time, making it ideal for scenarios where data needs frequent updates or adjustments. To utilize Canvas, users need Elasticsearch for data storage and Kibana for the user interface, with Metricbeat serving as a source of live data to demonstrate how data can be integrated into presentations. Users can create "workpads," which function like presentations, and add elements such as charts and graphs, using a variety of data sources including Elasticsearch Raw Documents and SQL syntax. The interface allows for easy customization and real-time data updates, ensuring that presentations remain current and engaging.

Elastic APM has introduced beta support for monitoring Go applications, focusing on measuring and reporting response times through a detailed agent that instruments application code. The agent uses an API to record transactions and spans, capturing the time taken for operations and their context, which are processed and sent to the Elastic APM Server for indexing in Elasticsearch. In designing the agent, minimizing overhead was crucial, leveraging Go's toolchain and libraries to reduce memory allocation and CPU usage. The agent processes data off the main code path using background goroutines and employs a buffered channel to handle data flow, discarding excess data when necessary. Failure designs ensure system reliability by incorporating a circular buffer to manage data when the APM Server is unavailable. Performance optimizations include object pooling and a custom zero-copy, zero-allocation JSON encoder to improve efficiency. Elastic's gobench tool is used to benchmark the agent's performance continuously, storing results in Elasticsearch for analysis and future improvements.

Elastic Stack's machine learning capabilities have been enhanced in version 6.4 to better handle changes in system behavior, particularly in identifying anomalies in time series data. The improvements focus on addressing concept drift, where historical data models may become inaccurate over time due to recurring, gradual, or sudden changes in the data. The updated system introduces parametric forms for detecting sudden changes, allowing for more efficient and robust model adaptation. This approach enables the model to seamlessly balance between adapting quickly to new data while maintaining stability and accuracy in predictions, particularly when sudden shifts occur, such as linear scaling or step changes. The enhancements allow for better anomaly detection by using a combination of historical data and real-time updates, making the model more resilient to unusual data intervals. Users can explore these advancements by trying out Elastic Stack's features with a 30-day trial or on Elastic Cloud.

Elastic has officially become a public company, listed on the New York Stock Exchange under the ticker "ESTC," marking a significant milestone in its journey since the release of Elasticsearch in 2010. The company was founded on the vision that search can transform interactions with data, by offering real-time results, scalability to query vast amounts of data, and relevance in providing actionable insights. Over six years, Elastic has achieved more than 350 million product downloads, a developer community exceeding 100,000 members, and a customer base of over 5,500, illustrating the broad applicability of its search technology in various fields, from pairing riders with drivers on Uber to enhancing cybersecurity operations at major organizations. As a public entity, Elastic remains committed to supporting its global developer community, enhancing the Elastic Stack with new features, and providing flexible deployment options. The company emphasizes its dedication to its foundational principles and the ongoing success of its users, customers, and partners.

In a recent update on the Kibana project, significant advancements were highlighted, including the merging of the Security Spaces feature into master and the ongoing development of granular application privileges as part of the security roadmap. The Canvas integration saw enhancements with the introduction of a persisted "Read Only Mode" and the completion of the first phase of ad-hoc grouping. Efforts in improving rollup data interaction within Kibana's Discover and Visualize features were also noted, along with the addition of a telemetry collector for rollup-based index patterns. The Kibana team is actively refining the K7 design, incorporating new app icons, a redesigned login, and improvements to the KQL query bar. Additionally, progress was made in platform testing, bug fixes, and the implementation of new features across various Kibana components, including the GIS App, which now supports dark theme, layer zoom levels, and configurable vector layer tooltips.

Elastic offers two managed search services, Elastic Site Search and Elastic App Search, both utilizing the Elasticsearch engine to enhance search experiences with minimal implementation time. Elastic Site Search is ideal for those who prefer an automated approach, using a Site Search Crawler to index web pages and offering a user-friendly dashboard for fine-tuning relevance and search customization, making it suitable for non-technical users. In contrast, Elastic App Search is API-centric, providing developers with programmatic control over search functionalities, allowing for deep customization and integration into applications, and offering advanced features like geolocation-based search and sophisticated analytics. Both services are designed to cater to a range of business needs, from e-commerce platforms to SaaS offerings, with Elastic Site Search focusing on automated, out-of-the-box solutions and Elastic App Search offering flexible, code-driven search capabilities.

Rightmove, the UK's largest property rental and sales website, transitioned from a monolithic search engine to Elasticsearch in 2014 to address scalability and functionality challenges. This shift was part of Project Odin, an internal initiative to find a more efficient search solution. Elasticsearch was chosen for its faster indexing, ease of use, scalability, and built-in features, including geo-search capabilities that were previously lacking. These capabilities enabled Rightmove to enhance its website with advanced mapping features like "Draw-a-Search" and "Where Can I Live?" tools, offering users personalized search experiences based on location and other criteria. Additionally, Elasticsearch's percolator feature allows Rightmove to efficiently match property documents to saved user queries, facilitating automatic notifications of matching properties. The transition to Elasticsearch also supports Rightmove's microservice development, enabling more frequent deployments and improving customer experience with enhanced search and reporting functionalities.