February 2019 Summaries

The Elastic APM Java Agent is a free, open-source tool designed to provide insights into application performance and analyze error root causes, supporting distributed tracing particularly useful for service-oriented architectures. While it inevitably introduces some overhead, the agent is designed to minimize impact on application performance through strategies such as running reporting processes on a background thread and reusing objects to reduce memory allocations. The agent's performance can be fine-tuned through configuration options, including adjusting the sample rate of requests recorded and sent to the APM Server, which affects network bandwidth and disk space usage. The agent's overhead in terms of latency is kept minimal by employing efficient data transfer techniques and by processing stack traces asynchronously. Additionally, users can optimize performance by managing the collection of headers, cookies, and stack traces.

In the article "Product Observability with the Elastic Stack" by Tanya Bragin, the author explores the concept of observability, explaining its significance in operational systems and its growing traction in the tech industry. Observability is described as an intrinsic system attribute that enables operators to detect and address undesirable behaviors in production environments, relying on a combination of metrics, logs, and application traces, often referred to as the "three pillars of observability." The article discusses the common challenges organizations face in collecting and making this data actionable, highlighting the benefits of integrating these data streams into a single operational store to improve efficiency and reduce costs. Elastic Stack, or ELK Stack, is presented as a versatile solution for centralizing various types of operational data, including logs, metrics, and traces, and is particularly noted for its adaptability to emerging requirements like anomaly detection. The piece emphasizes how Elastic's tools, such as Metricbeat and Elastic APM, help streamline observability efforts by automatically correlating different data types, thus facilitating more effective monitoring and troubleshooting in complex environments like those using Kubernetes.

The article by Medcl Zeng outlines a practical project that combines a Raspberry Pi with Elasticsearch to create a real-world alarm system capable of notifying users of critical server issues through both visual and auditory alerts. By leveraging Raspberry Pi's GPIO pins and a relay module, the setup can control a high-voltage warning light and a buzzer, providing an immediate and noticeable alert when server downtime is detected. The system integrates with Elasticsearch's alerting feature, using a custom webhook service to trigger the alarm in response to data from the Heartbeat monitoring tool, ensuring the alert is raised whenever a service goes down. This innovative approach to infrastructure monitoring bridges the digital and physical realms, offering a robust solution for capturing attention in critical situations.

In February 2019, significant developments were underway for Kibana, focusing on platform enhancements, security updates, and new features across various teams. The new platform team had its official kick-off, concentrating on modifying existing platform components and planning plugin migrations. Security efforts were directed toward integrating feature controls and small changes to the Content Security Policy for compatibility with Kibana 6.7. Concurrently, the Middleware team worked on the Secret Service, actions service design, and created an RFC proposal for review. The Geo Maps team made strides towards making maps embeddable in dashboards and started localization efforts, while the Elastic Maps Service began data production of tile-data. The Canvas team tackled performance issues by optimizing React rendering and began simplifying state management, aiming for improved user interactions and performance. The design team focused on dark mode refinements and component updates, while operations prioritized resolving upgrade process blockers. The KibanaApp team dedicated efforts to stabilizing the 7.0 release by addressing bugs and enhancing user interface elements, with notable improvements in visualization metadata support and query handling, alongside ongoing TypeScript conversion.

CenturyLink, the third-largest telecommunications company in the United States, has developed a robust network management framework using open-source technologies to handle its vast and evolving data needs. As the company expanded through mergers and acquisitions, it faced challenges in managing data from over 300,000 devices and generating over 1 million alerts daily. Initially relying on a basic network management system architecture, CenturyLink experienced frequent outages and database performance issues as their operations grew more complex. Transitioning to a Command Query Responsibility Segregation (CQRS) architecture and integrating Elasticsearch significantly improved query speeds and system scalability, reducing downtime and increasing operational efficiency. The adoption of Elasticsearch, along with other open-source tools like Redis and Kafka, facilitated seamless incorporation of new network management systems and devices. CenturyLink's Service Assurance team now aims to leverage the Elastic Stack to create a comprehensive framework that utilizes predictive algorithms and operationalizes diverse data types, including network utilization and weather, to produce actionable insights for better decision-making.

The Department for Work and Pensions (DWP) in the UK, responsible for welfare, pensions, and child maintenance policy, implemented a Digital Experience Monitoring (DXM) project to enhance the performance monitoring of its Siebel system. The initiative aimed to provide the Live Service Support team with more proactive capabilities in ensuring system performance by utilizing the Elastic Stack for log parsing, data visualization, and real-time analysis. An internal squad was formed in 2017, working in Agile sprints to develop the architecture through discovery, alpha, beta, and live phases. The Elastic Stack, comprising Logstash, Elasticsearch, and Kibana, was chosen for its scalability, cost-effectiveness, and ability to deliver insights into system performance and issues. The DXM system processes data from various sources, enabling the DWP to identify and solve potential performance concerns before they affect users. The success of this project, which includes creating dashboards for performance visualization and automation, has been acknowledged by senior leaders and shared across the DWP to support similar monitoring solutions. Upgrades to the Elastic Stack are underway to incorporate machine learning for anomaly detection.

Mayr-Melnhof Group, a major cardboard producer based in Vienna, Austria, utilizes the Elastic Stack to manage and optimize their complex production process, which involves producing over 1.7 million tons of cardboard annually. The company deployed over 25,000 sensors to monitor various parameters, such as pressure, temperature, and machine speed, generating around 2 TB of data. Initially lacking the tools to process this data efficiently, they adopted Elasticsearch, Kibana, and Logstash to visualize and analyze production data in real-time, leading to a 20% reduction in the consumption of costly materials within four months. This integration of Information Technology (IT) and Operational Technology (OT) enhanced their ability to promptly address deviations and optimize production.

On February 21, 2019, Elastic Engineering addressed the runc vulnerability affecting older versions of Docker, which could potentially impact Elastic Cloud Enterprise (ECE) users by allowing attackers to gain administrative privileges through compromised containers. Although the Elasticsearch Service, based on ECE, remains secure with continuous monitoring and patching, ECE users are advised to upgrade their Docker engine and monitor runc binaries to mitigate risks. Elastic emphasizes that ECE incorporates multiple security measures to prevent privilege escalation, including running the Elastic Stack as a non-root user and restricting container modifications. The company recommends that users running ECE 2.x upgrade to the latest Docker version, while those on ECE 1.x should move to ECE 2.1, which is compatible with newer Docker versions. Elastic offers support and consulting services to assist users in managing the upgrades and addressing any concerns regarding the vulnerability.

Elasticsearch 7.0 introduces new tools to simplify relevance tuning, a challenging task often hindered by the need for extensive human-annotated training sets. The update includes the addition of the rank_feature and rank_features fields, which improve relevance metrics by facilitating efficient ranking queries based on non-textual signals like popularity or authority. These fields support top-k retrieval optimizations, allowing for quicker retrieval of top matches without sacrificing performance. Additionally, the Script Score Query enhances flexibility by allowing users to define custom scoring formulas through Painless scripting, accommodating various relevance signals such as numeric, geo, or vector fields. These innovations aim to streamline the process of improving search relevance while maintaining query performance, with the promise of future developments for dynamic features like recency or geo-distance.

John Deere has transitioned from using the ELK Stack to Elastic Cloud Enterprise to enhance its data management capabilities, crucial for modern farming operations. Originally utilizing JDLink to gather data like geographic location and machine health on farm equipment, the company faced challenges in scaling as the application expanded. The Intelligent Solutions Group (ISG) at John Deere adopted the Elastic Stack in 2013, allowing them to aggregate logs and visualize systems holistically, which improved the management and analysis of application data. Over time, John Deere's platform has evolved to support various farming needs, including remote management and field management, utilizing Elasticsearch to scale their infrastructure significantly. Currently, John Deere's Logcentral@Deere system handles vast amounts of data across numerous applications, leveraging Elastic Cloud Enterprise for features such as integrated authentication, monitoring, and machine learning, with plans to further enhance data processing capabilities in the future.

As enterprises increasingly adopt microservice architectures, the need for effective monitoring tools like distributed tracing becomes essential to manage the complexity and performance of these systems. Distributed tracing, as part of the three pillars of observability alongside logging and metrics, helps track latency and conduct root cause analysis across microservices. Elastic APM, built on the Elastic Stack, offers real-time application performance monitoring by collecting detailed data on requests and transactions, supporting distributed tracing with OpenTracing compliance. Despite challenges with standardization and interoperability among different tracing systems, initiatives like the OpenTracing specification and W3C Trace Context aim to create unified APIs and formats for better integration. Elastic APM facilitates compatibility through its OpenTracing bridge, allowing data from various tracers to be imported into its platform for comprehensive analysis, leveraging Elasticsearch's scalability for storing and visualizing tracing data. Additionally, Elastic APM Real User Monitoring captures client-side performance metrics, providing insights into real user experiences.

KPN Security Services, a SOC/SIEM security management company from the Netherlands, has harnessed the Elastic Stack to tackle the challenges of managing rapidly growing and complex data volumes while maintaining its commitment to preventing and responding to threats. At an Elastic{ON} Tour event in Amsterdam, KPN representatives Han Pieterse and Marius Iversen discussed how the Elastic Stack helps mitigate "security data overload" by enhancing data analysis, visibility, and management, enabling the 400-member security team to focus on core security functions. Before adopting the Elastic Stack, KPN's single-tenant, manually configured setup limited scalability, but the integration of the ArcSight Logstash plugin and multiple data pipelines through Kibana dashboards simplified data ingestion and improved efficiency. KPN plans to leverage the Elastic Stack's machine learning capabilities for security and non-security purposes, such as monitoring server activities for breaches, detecting anomalies in financial transactions and employee behavior, and forecasting resource requirements to prevent memory shortages.

The Kibana weekly update for February 2019 highlights several developments, including the imminent release of version 7.0, featuring new themes and enhanced functionality such as Elastic Charts, which will replace existing chart libraries. Security improvements include a new content security policy and updates to the Role Management UI, alongside the removal of deprecated features. Localization efforts are advancing with Chinese translations, and significant progress is being made in middleware and alerting features. The platform is transitioning to a new Elasticsearch service, with the canvas team leading the migration due to their flexible plugin system. Numerous design and visual testing improvements are underway, including the use of Percy and Applitools for visual testing. The design team is focused on polishing the UI, incorporating feedback, and cleaning up applications, while the data production and operations teams are preparing for a robust 7.0 release. Additionally, enhancements to the KibanaApp, including responsive dashboard layouts and the introduction of TypeScript in the saved object client, are being implemented.

Alex Marquardt provides a detailed guide on how to debug Elasticsearch source code using IntelliJ IDEA, specifically for version 6.6. The process involves downloading the Elasticsearch source code from GitHub, configuring it with the Gradle build system, and setting up the Java environment variables necessary for building the project. The guide outlines steps for importing the Elasticsearch project into IntelliJ IDEA and starting the project in debug mode, which allows users to set breakpoints and step through the Elasticsearch and Lucene code. This setup is designed to help developers understand the inner workings of Elasticsearch, though it is noted that the instructions are specific to older versions and will not work for version 7.5 and later due to changes in the codebase. The blog post emphasizes the utility of reviewing accompanying text files like CONTRIBUTING.md and TESTING.asciidoc for additional guidance and encourages users to consult IntelliJ IDEA documentation for debugging assistance.

The Elastic Common Schema (ECS) is introduced as a new open-source specification designed to standardize data structuring in Elasticsearch, enhancing both interactive and automated data analysis across diverse sources. ECS aims to resolve formatting inconsistencies by providing a uniform set of document fields, enabling seamless data integration from various environments and vendors. This facilitates more efficient search, visualization, and machine learning-driven anomaly detection. ECS's taxonomy, organized into core, extended, and custom fields, allows for predictable data modeling while supporting custom use cases. By adopting ECS, users can streamline data correlation, reduce duplication, and ease the development of analytics content, thus improving interoperability and efficiency in data management. Despite its benefits, implementing ECS requires effort, particularly for those familiar with Elasticsearch index templates and Logstash. The schema, available on GitHub and licensed under Apache 2.0, is in Beta2 and expected to become generally available, with future updates aimed at expanding use case support.

The blog post provides updates on various modules within the Elastic Beats framework, including Filebeat, Metricbeat, and other components that are part of the Elastic Stack. A significant highlight is the introduction of a new sub-command in Filebeat that facilitates the generation of necessary files for new modules or filesets, enhancing customizability and flexibility. Changes to the Filebeat registry have been made to improve future migration processes, while updates to the Central Configuration aim to better manage a large number of Beats and configurations. Additionally, the Stack Monitoring team has made strides in enabling log displays within the Stack Monitoring application, allowing for the integration and parsing of various Elasticsearch logs. The document also details recent bug fixes, code refactoring, and enhancements across different Beats, with the objective of aligning them more closely with Elastic Common Schema (ECS) standards. Overall, the updates aim to improve the functionality, reliability, and scalability of the Elastic Beats system.

The announcement highlights the preview release of the Elastic APM .NET agent, designed to collect user feedback and demonstrate progress within the community. This agent, part of the Elastic Application Performance Monitoring solution, provides insights into application performance, error tracking, and end-user browser experience, and is available for various programming languages, including Java, Node.js, Python, Ruby, JavaScript/RUM, and Go. The preview version features auto-instrumentation for ASP.NET Core 2.x, Entity Framework Core 2.x, and outgoing web requests via the HttpClient class on .NET Core, and offers a Public Agent API for manual instrumentation on other frameworks. Distributed as NuGet packages, it includes Elastic.Apm.All, Elastic.Apm, Elastic.Apm.AspNetCore, and Elastic.Apm.EntityFrameworkCore packages, with the ability to configure the agent using environment variables or the IConfiguration interface. Users are encouraged to provide feedback through discussion forums or GitHub, with the understanding that the release is subject to changes based on community input and ongoing development, which may include features like distributed tracing and support for ASP.NET Classic.

Andreas Helmer, a data enthusiast from Ontario, Canada, shares his journey of managing Type 1 Diabetes using the Elastic Stack, a suite of open-source tools designed for data analysis. After experiencing a hypoglycemic incident that highlighted the need for better diabetes management, Helmer adopted an insulin pump and continuous glucose monitor (CGM) to track his blood sugar levels more accurately. By utilizing Logstash, Elasticsearch, and Kibana from the Elastic Stack, he was able to consolidate and analyze data from various devices, creating customizable dashboards to monitor his glucose levels in real-time. The integration of machine learning within Elastic further enabled him to forecast blood sugar trends with surprising accuracy, improving his ability to stabilize his glucose levels. While Helmer stresses that his approach is a personal experimentation rather than a medical endorsement, he highlights the potential of Elastic tools to transform personal health data into actionable insights, thereby enhancing his quality of life and providing peace of mind.

The Elastic Search Awards, formerly known as the Elastic Cause Awards, recognize innovative and transformative uses of the Elastic Stack across various sectors, including philanthropy, business evolution, and unique technological applications. This year, more than 30 organizations and individuals in the Americas have applied, showcasing projects that range from protecting the elderly from phone scams to developing a visual search system for e-commerce. The awards are divided into categories such as Cause Awards for projects with measurable societal impact, Cluster Awards for technology innovation, and Search! Awards for business transformation. The judging criteria focus on impact, originality, risk-taking, sustainability, value, and collaboration. The winners will be announced at the Elastic{ON} Tour San Francisco on February 28, 2019, and will receive various prizes, including training subscriptions and event admission. Applications for the Europe, Middle East, Africa, Asia Pacific, and Japan regions remain open for later in the year.

In this blog post, Molly Struve, a Senior Site Reliability Engineer at Kenna Security, outlines various techniques used to enhance search performance in their Elasticsearch cluster, which manages over four billion documents. By organizing data into client-specific indexes, the company reduced the number of shards queried, thereby speeding up searches. Emphasizing the importance of using filters over queries to reduce computational load, Struve shares insights from an upgrade to Elasticsearch 5.0, which highlighted the efficiency of filters. Additionally, the company improved search speeds by storing IDs as keywords rather than integers, resulting in a 30% performance increase, and by restricting complex user-generated queries that could burden the system. These strategies have contributed to making Kenna's Elasticsearch cluster more stable and scalable, as they continue to support a growing volume of data.

In the blog post, Dave Moore discusses the limitations of sorting products solely by average ratings in online retail environments, which often results in a poor user experience and diminished sales. He introduces an alternative approach using Elasticsearch to calculate and sort products by the Wilson score, a method that balances ratings with the number of reviews to provide a more accurate reflection of product quality. The post provides a step-by-step guide on implementing this sorting method using custom scripts in Elasticsearch, highlighting the efficiency and ease of execution. By using the Wilson score, retailers can ensure that products with the most reliable ratings appear at the top of search results, facilitating faster decision-making for consumers and potentially boosting sales. Moore encourages search engineers to adopt this method to enhance search relevance and improve customer satisfaction, emphasizing the low cost and significant benefits of its implementation.

In 2019, Elasticsearch introduced the WAND algorithm to enhance the speed of retrieving top hits in search queries, building on earlier work with the MAXSCORE algorithm. Originally presented by Stefan Pohl, MAXSCORE aimed to optimize search by skipping non-competitive documents, but its integration was hindered by the need for static indexes. The implementation of WAND, which uses a more refined approach by assigning weights to query clauses, allowed for dynamic index optimization, significantly speeding up query execution. Further improvements were made by adopting block-max WAND, a variant that handles outlier scores more efficiently by splitting postings into fixed-size blocks and recording maximum impact scores per block. This development led to remarkable speedups in various query types, though it introduced changes such as potentially less accurate total hit counts and the requirement for non-negative scores. These optimizations are set to be incorporated into Lucene 8.0 and Elasticsearch 7.0, offering substantial performance benefits while allowing users to adjust the number of hits counted to balance speed and accuracy.

Kenna Security, a company that helps Fortune 500 clients manage cybersecurity risks by analyzing data on assets and vulnerabilities, has significantly improved its Elasticsearch indexing capabilities to handle the substantial growth in data volume. Initially struggling with a cluster that was unstable and inefficient, Kenna's cluster now holds 4 billion documents and processes over 200 million daily. Key techniques that facilitated this transformation include adjusting the refresh interval to optimize resource allocation, implementing bulk processing to efficiently handle large batches of documents, and using routing to reduce the number of threads needed for indexing by grouping documents by shard. These strategies have enabled Kenna to maintain efficient indexing and search capabilities as their data demands increase. The improvements came with careful planning and adaptation, preparing Kenna for continued scaling, and ensuring their clients can effectively organize and access their data. Molly Struve, a Senior Site Reliability Engineer at Kenna, played a pivotal role in this successful scaling process over her three years with the company.

In February 2019, Elasticsearch Service announced a restructuring of its pricing model, introducing separate charges for snapshot storage and data transfer costs, which were previously discounted. Snapshot storage costs are based on the size and number of API requests made to store backup snapshots, while data transfer costs pertain to data moving into, out of, and within Elasticsearch deployments. The service provides free allowances for both snapshot storage and data transfer, with charges applied only when usage exceeds these thresholds. The new pricing model aims to give users more control over their expenses by detailing and metering these specific costs. The changes were initially introduced as a zero-cost line item in invoices to estimate future charges, with actual billing starting from March 2019. Existing annual customers will not see changes until contract renewal, while new customers are subject to the updated pricing. The changes apply across all cloud providers, ensuring a uniform rate structure.