September 2019 Summaries

Kibana's recent updates highlight significant progress across various components and features, including the introduction of the new Lens visualization, which is now part of the Visualize app and ready for user feedback. The platform team has made strides in migrating functionalities to the New Platform, resolving several bugs, and completing crucial audits without identifying new blockers. Enhancements to the Geo-Maps app include the ability to cancel outdated Elasticsearch requests and the transition from Mapbox tooltips to EuiPopover for improved usability. The App Architecture team advanced the migration of services to the new platform and continued refactoring visualization types, while the Elastic Charts team worked on improving the library, introducing new features, and progressing with pie chart implementation. Additionally, the Graph and Discover teams have been active in de-angularizing components, and the Canvas team focuses on localization. Accessibility remains a priority, with new resources available for building and testing with EUI, while updates to the EuiDataGrid enhance data formatting and readability. The dashboard workflow is undergoing refinement to better integrate with various applications, and stack services like alerting have received performance and functionality improvements.

Financial services companies are increasingly leveraging the Elastic Stack to address various challenges and regulatory requirements, as highlighted by Elastic's solutions architect, Michael Down. The Elastic Stack, known for its logging, search, and security analytics capabilities, is being used by organizations like Citi, Barclays, and USAA for tasks ranging from compliance with PSD2 regulations to fraud monitoring and trade tracking. The technology offers advantages such as scalability, ease of management, and parallel operation alongside legacy systems, making it attractive for adapting to new regulations like Open Banking and CSDR. The Stack's horizontal scalability and API-first approach allow financial institutions to innovate without significant investment in expensive technology, enabling real-time data access and processing. This adaptability is crucial as financial services firms face evolving regulatory landscapes, requiring them to manage large volumes of data efficiently and to ensure timely settlements. Elastic's machine learning features further enhance trade tracking by providing early warnings for potential settlement issues, underscoring its value in modernizing financial services infrastructure.

On September 24, 2019, Elastic announced the release of the first stable versions of their PHP clients for Elastic Site Search and Elastic App Search. These PHP clients provide developers with tools to integrate powerful search functionalities into their applications using Elastic's search solutions. The Site Search PHP client is known for its ease of use, out-of-the-box flexibility, and robust web crawler, allowing for the quick setup of engaging search experiences supported by the Elastic Stack. Meanwhile, the App Search PHP client offers a suite of dynamic APIs that enable comprehensive search integration within applications, featuring tools like Synonyms, Relevance Tuning, and Curations for optimized search performance. Both clients can be installed using Composer, and they facilitate the creation, management, and optimization of search engines in multiple languages. Elastic offers a 14-day free trial for users to explore these solutions.

The text discusses how the Elastic Stack can be utilized to enhance observability for MuleSoft CloudHub by integrating its monitoring data with other IT systems, thereby offering a unified view of logs and metrics across the organization. This integration reduces the need for multiple monitoring interfaces and enables advanced alerting and visualization capabilities using Kibana dashboards. The Elastic Stack allows for long-term data retention and real-time analytics by synchronizing CloudHub data, including logs, API events, and worker metrics, with Elasticsearch. The article provides a detailed guide on using Logstash for data synchronization, including configuration and pipeline management, and highlights the benefits of centralizing Logstash management through Kibana. Sample visualizations for CloudHub data are provided in the GitHub repository, making it easier to monitor CloudHub services by leveraging Elastic's robust features.

Stuart Cam and Russ Cam discuss the challenges and solutions of upgrading Elasticsearch servers and codebases, particularly when using different major versions of the NEST client and Elasticsearch server. They explain the compatibility issues between client and server versions, noting that using a 6.x .NET client with a 7.3 Elasticsearch cluster is not recommended due to differences in request and response object formats. To solve this, they propose using namespaced versions of NEST and Elasticsearch.Net clients available through their continuous integration (CI) server, allowing developers to reference different client assemblies tied to specific major versions. By replacing official Nuget packages with those from the CI package source and prefixing existing types, developers can use both the 6.x and 7.x clients simultaneously in the same project, facilitating a gradual migration to newer versions. The authors stress that this approach is temporary and should be used to aid the transition, as CI packages do not offer the same level of support as official releases. The article also touches on the use of AssemblyRewriter in their CI process to rewrite assemblies for performance benchmarking, emphasizing the utility of these packages for other users facing similar upgrade challenges. They conclude by recommending the Elasticsearch Service on Elastic Cloud for a straightforward upgrade path.

In 2019, Elastic Stack focused on enhancing its alerting capabilities by introducing a new framework integrated into Kibana, aiming to improve user experience across its diverse product range. Elastic's alerting system, initially manifested through Watcher, has evolved based on user feedback, highlighting the need for robust alerting mechanisms that are deeply integrated across different use cases like SIEM, APM, and Uptime. The new framework, part of the 7.11 release, features improved observability, sophisticated detection, and action capabilities, and supports integration with third-party platforms like Microsoft Teams alongside existing ones like Slack and PagerDuty. Elastic Stack's alerting system now incorporates a foundational Task Manager for scalable task management and APIs for customizable alerts and actions. This development is part of a phased rollout that includes the creation of a comprehensive user interface in Kibana, aiming to make alerts first-class entities that enhance system observability and enable complex detection and response actions, ultimately providing a more intuitive and integrated user experience across its products.

Elastic has announced its compliance with several important information security standards, including ISO/IEC 27001:2013, 27017:2015, and 27018:2019, as well as the completion of the Cloud Security Alliance (CSA) Security Trust and Risk (STAR) certification. These certifications, conducted by independent third-party auditors, underscore Elastic's commitment to maintaining industry-leading best practices in information security and privacy. The ISO 27001 certification involves implementing a comprehensive security program through an Information Security Management System (ISMS) that assesses risks and applies security controls. ISO 27017:2015 offers guidance for cloud service security controls, while ISO 27018:2019 ensures protection of personally identifiable information (PII) processed by cloud providers. The CSA STAR certification evaluates the security of cloud service providers like Elastic, combining ISO/IEC 27001:2013 standards with the CSA Cloud Controls Matrix, reflecting Elastic's dedication to securing customer data across its services.

In the update for Kibana, significant progress was made across various areas, including the New Platform migration, where flexible planning is enabling rapid reprioritization for efficient plugin migration. Notable advancements include the review of space-specific default routes, adjustments to SAML redirect mechanisms due to RelayState limitations, and ongoing work on the TaskService RFC and SavedObjectsService. The Maps team released EMS Landing Page v7.4 with improved zoom capabilities and HTTPS redirection, while the App Architecture segment saw enhancements in data access services and the splitting of data and expressions plugins. Efforts in the Kibana App focused on bug fixes, new features such as KQL saved queries in Maps, and a move towards React and TypeScript for the search bar and field manager. The Canvas team is prioritizing localization, while the Design team introduced updates to compressed forms and the datagrid component, improving form usability and automatic data schema detection. Lastly, the Reporting and Alerting teams concentrated on UX improvements, alert creation at system startup, and TypeScripting to streamline future developments.

In a blog post by Stuart Cam, the process of building a real-time address search system using the Australian Geocoded National Address File (G-NAF) and Elasticsearch is discussed. The G-NAF, an authoritative and comprehensive dataset containing nearly 14 million addresses, was made publicly available by PSMA Australia in 2016. The blog outlines how to leverage Elasticsearch's search_as_you_type feature to implement a typeahead search functionality, demonstrating step-by-step procedures such as downloading necessary tools, staging the G-NAF datasets into a SQL database, creating Elasticsearch indices and mappings, and conducting both address and geo-distance queries. The tutorial is framed within the context of the author's experience with Elastic's Spacetime program, which allows engineers to explore innovative projects, in this case utilizing the F# programming language.

In version 7.5, APM Server introduced changes to its Index Lifecycle Management (ILM) policies, allowing for greater customization in managing Elasticsearch indices as they age. This feature, initially made available in Elasticsearch 6.7, categorizes indices based on their usage from hot to cold, with management actions defined by factors such as time and performance. APM Server now creates specific templates and ILM policies for different event types like transactions and errors, enabling tailored management of data. Default policies include phases like hot and warm, ensuring indices are efficiently managed without automatic deletion, though customization options allow for additional phases such as delete. Users can modify ILM policies through Kibana, and configurations are checked every 10 minutes by default, with the option to adjust this interval. APM Server on Elastic Cloud also supports ILM and can implement hot-warm-cold architectures, with future plans for more flexible policy options. The document emphasizes the importance of updating policies and templates when upgrading to newer versions, and invites feedback from users on their experiences.

Elastic App Search offers a versatile, API-centric search engine that enables users to integrate search functionalities into various applications, ranging from ecommerce to mobile apps, with ease. It comes as both a managed service and a self-managed distribution, with the latter available for free under a basic Elasticsearch license. The platform's newly introduced Result Settings enhance search performance by allowing users to customize and optimize query responses, such as highlighting query matches and trimming unnecessary data to deliver concise, rich results promptly. By enabling snippet fields and setting character limits, users can ensure search results are both efficient and relevant, thus reducing strain on search engines and improving user experience. These tools are designed to seamlessly support both UI-based and API-driven interactions, catering to a broad spectrum of developer needs.

Elasticsearch Service has expanded its availability on Google Cloud Platform (GCP) by launching in the Sydney region, marking its second GCP region in the Asia Pacific and sixth globally. This expansion enables faster and more responsive solutions for customers in the APAC region while maintaining service parity with other global locations. Existing users can immediately leverage the new region, while new users can explore the service through a free 14-day trial. The expansion aligns with an ongoing partnership between Elastic and Google Cloud, which includes future plans to introduce additional GCP regions, enhance billing integration, and improve native GCP Console interactions for a seamless Elastic Stack experience. The launch follows the recent introduction of the Tokyo region and comes as part of Elastic's commitment to providing comprehensive support and an optimized user experience for GCP customers.

Elasticsearch is a versatile tool commonly used for data search and analytics across various applications, such as logging and security analytics. To utilize its capabilities, data must first be ingested into Elasticsearch Service, which can be hosted on Elastic Cloud or on-premises. Several methods exist for data ingestion, including Elastic Beats, Logstash, language-specific clients, and Kibana Dev Tools. Elastic Beats are lightweight data shippers ideal for resource-constrained environments like IoT devices, while Logstash offers robust data processing capabilities, albeit with higher resource requirements. Language clients provide integration options within custom applications, and Kibana Dev Tools offers a powerful interface for developing and debugging Elasticsearch requests. Each method leverages Elasticsearch's RESTful API, enabling users to choose the best approach based on their specific needs and environments.

Kibana's recent updates include significant developments across various aspects of the platform, with a focus on enhancing functionality and migrating to the New Platform, aimed for completion before the release of version 8.0.0-alpha1. Notable improvements include the implementation of security-specific default routes and efforts to address SAML redirect issues through intermediary pages and relay state integration. The team is actively working on sharing saved objects across multiple spaces, with ongoing investigations and documentation. The migration to the New Platform is a priority, with teams receiving guidance on plugin transitions and addressing blockers. Progress in application architecture involves changes to embeddable action interfaces, and advancements in visualization plugins, TypeScript conversion, and stabilization of tests. Efforts in operational efficiency have led to a dramatic reduction in CI run workers and improvements in filesystem performance. Additionally, the Elastic Charts migration continues, alongside updates in Discover and Canvas i18n, and the design team is enhancing Kibana's interface through EUI integration. Reporting functions are being refined with a focus on Chromium stability, scheduled reports, and telemetry is transitioning to open-source. Lastly, alerting improvements include network call white-listing and developments in UI for alerting and actions are underway.

The blog post details the process of deploying a sample application and Metricbeat in a Kubernetes environment managed by Elastic Cloud on Kubernetes (ECK) to facilitate data ingestion into an ECK-managed Elasticsearch cluster. It covers the instrumentation of a basic Node.js application with Elastic APM, including necessary configurations like obtaining the APM server URL and token. The post also explains how to expose the application and access it locally, while guiding users on setting up the APM UI to monitor transactions and metrics. Additionally, it describes deploying Metricbeat to collect and send metrics data to the Elasticsearch cluster securely, highlighting the importance of using ConfigMaps and Kubernetes secrets for SSL and connection configurations. The blog emphasizes that ECK, developed by the creators of the Elastic Stack, is an effective solution for securely managing and observing applications on Kubernetes.

Kudos provides a free platform designed to help researchers enhance their communication strategies and reach broader audiences beyond traditional academic circles. By allowing researchers to create pages for their publications in plain language and track engagement metrics, Kudos bridges the gap in communicating research impact effectively. Facing limitations with their initial MySQL database for handling extensive data, Kudos transitioned to the Elastic Stack, which offered scalable solutions for time series data storage and visualization through Kibana and data ingestion via Logstash. The Elastic Stack enabled Kudos to integrate with publisher Manuscript Submission Systems, facilitating the tracking of manuscripts from submission through publication. The implementation of Elasticsearch Service on Elastic Cloud minimized operational overhead and allowed easy scaling, benefiting both the engineering and account management teams by providing real-time data insights and flexible visualization capabilities.

Shay Banon details a lawsuit filed by Elastic against floragunn GmbH, developers of the Elasticsearch security plugin Search Guard, for allegedly copying source code from Elastic's proprietary security features. The issue was discovered after Elastic made its code publicly accessible, revealing a history of what Elastic describes as intellectual property theft by floragunn, involving decompiling binary releases to copy code before it was opened. Elastic filed the lawsuit in the United States District Court for the Northern District of California and issued a DMCA takedown notice to prevent the distribution of the infringing code. Elastic informs Search Guard users that they might be running infringing code and highlights that Elasticsearch now includes free security features to avoid the need for unprotected clusters, urging concerned users to reach out for support.

Elastic Cloud on Kubernetes (ECK), though still in its alpha stage, has attracted significant interest from both the Elasticsearch and Kubernetes communities as it offers a streamlined approach to deploying the Elastic Stack on Kubernetes. Developed by the creators of the Elastic Stack, ECK simplifies the deployment and management processes, making it possible to launch ECK and the Elastic Stack with minimal commands using tools like Minikube or Google Kubernetes Engine (GKE). The blog provides a detailed guide on setting up ECK, deploying the Elastic Stack, and scaling and upgrading Elasticsearch within a Kubernetes cluster, emphasizing the ease of use with commands that ensure security and efficient configuration. By leveraging Kubernetes' capabilities, ECK facilitates zero-downtime scaling and upgrading, offering a seamless experience for users. The introduction of security features, such as default TLS and basic authentication, enhances the deployment's safety, while customization options like pod templates and virtual memory configurations allow for flexible management. The blog also highlights the capabilities of ECK in handling tasks such as deploying Metricbeat and a sample application instrumented with Elastic APM, which will be further explored in the second part of the series.

Elasticsearch Service on Elastic Cloud is now available on Microsoft Azure, offering users the convenience of a fully managed Elasticsearch service on their preferred cloud platform. This extension is part of a broader collaboration between Elastic and Microsoft to enhance user experience and cater to organizations standardized on Azure. The service allows for quick deployment of Elasticsearch and Kibana, offering features like Elastic APM, SIEM, and machine learning, along with regular updates and security patches. During its public beta, available in East US 2 and West Europe regions, users can benefit from free data transfer and storage, with technical support provided. The integration leverages Microsoft's VM capabilities to optimize various use cases and signifies a shared commitment to developer choice and managed services. Existing users can migrate seamlessly, and new users can begin with a free 14-day trial, reflecting a strategic alignment to meet critical application needs in search, logging, observability, and security.

Deloitte's Cyber Intelligence Centre (CIC) has adopted the Elastic Stack to enhance its multitenant cloud-based Managed Security Service Provider (MSSP) platform by leveraging its capabilities for cyber data lakes and threat hunting. This collaboration, which began in 2013, has evolved significantly, with Deloitte utilizing Elastic's machine learning features to detect new threats and enhance their cybersecurity offerings. In 2019, Deloitte and Elastic signed a multi-year Managed Service Provider Agreement to ensure long-term access to Elastic's commercial features, supporting the CIC's goal of expanding its threat analytics platform globally. The partnership has allowed Deloitte to integrate Elastic into their ecosystem of detection and response capabilities, including SIEM and security orchestration services, and aims to deliver new insights by combining data from various domains, ultimately reducing cyber risk for their clients.