June 2019 Summaries

Elastic Stack 7.2.0 has been made available on the Azure Marketplace through an Azure Resource Manager (ARM) solution template, allowing users to utilize its features, such as Elastic SIEM, within the Azure platform. The partnership between Elastic and Microsoft has enhanced the capabilities of the ARM template, which now supports the deployment of large Elasticsearch clusters with 32TiB managed disks and offers options like accelerated networking and new VM SKUs, including the Lsv2-series. The ARM template also introduces features like shard allocation awareness, multiple Logstash instances, shared storage for snapshots, and built-in user configuration to streamline Elastic Stack deployment on Azure. Furthermore, an upcoming Elasticsearch Service on Elastic Cloud will soon be available on Microsoft Azure, providing hosted Elasticsearch and Kibana clusters with features like one-click upgrades and Elastic support.

Metricbeat, with its 7.2 release, has expanded its capabilities to report metrics on Linux software RAID, providing detailed insights into RAID devices created with mdadm. These devices offer various metric interfaces, including ioctl, /proc/mdstat, and the /sys/block/mdX sysfs interface, with Metricbeat utilizing the latter for data collection. To begin monitoring, users need to enable the RAID metricset in Metricbeat's system module configuration and ensure proper setup if running within Docker. Once configured, Metricbeat categorizes disk states into active, total, spare, and failed, and reports these to Elasticsearch, allowing users to identify issues like failed disks using Kibana's Discover app. This integration offers a straightforward approach to monitoring Linux software RAIDs using the Elastic Stack, with support available through a free trial or community forums.

Elastic emphasizes the importance of authenticity in its interview process, encouraging candidates to be themselves as the company values diversity among its workforce. During interviews, Elastic aims to understand applicants as much as it wants them to understand the company, highlighting the diverse backgrounds of its employees, known as Elasticians. Key advice from current employees includes being problem-solving oriented, eager to learn, and passionate, as well as taking the time to familiarize oneself with the interviewers. Elastic invites potential candidates to explore open positions and learn more about the company's culture through its blog.

Elasticsearch improved its ability to store dates with nanosecond resolution starting from version 7.0 by replacing the outdated Joda-Time library with the Java Time API, which supports higher precision. This transition required significant code refactoring to ensure backward compatibility and maintain performance, particularly in indexing throughput and aggregation latency. The solution involved creating a new field mapper, date_nanos, to store dates as long values in nanoseconds despite the reduced date range up to the year 2262. Challenges arose during the migration, such as performance degradation due to exceptions and increased garbage collection times, which were addressed through optimization techniques like avoiding exceptions in date parsing and leveraging microbenchmarks for performance validation. Additionally, backward compatibility with existing Joda-Time-based methods was maintained by introducing a dual-format compatibility test and deprecating old methods with clear migration paths. Future updates aim to enhance the functionality across the Elastic Stack, including Kibana and Beats, for seamless integration and querying of nanosecond-precision timestamps.

Kibana 7.2 introduces feature controls, allowing administrators to customize which features are available to different users and spaces, enhancing the application's usability and security. Previously, feature access was managed by disabling features globally through the kibana.yml file, but now feature controls enable more granular management using role-based access control. This capability lets administrators tailor the Kibana experience by hiding unnecessary features in specific spaces, and securing access based on user roles, thus offering fine-grained access policies. The introduction of custom privileges allows for defining specific feature access, either globally or per space, with roles being additive in nature. This means users receive cumulative privileges from all assigned roles, requiring careful management to avoid contradictions. The combination of spaces and security offers a flexible approach to managing feature availability, ensuring features are only accessible when both visible in the space and authorized for the user. Kibana encourages users to explore these new controls and provides resources for feedback to continue improving the user experience.

Elastic has introduced a new consulting package called the Cloud Launch package, designed to expedite the deployment of cloud clusters on Elastic Cloud, which includes services like Elasticsearch, Elastic App Search, and Elastic Site Search. This two-day virtual consultancy provides configuration and launch assistance, aiming to accelerate project timelines and ensure successful deployment without the need for formal project initiation or reports. Offered in eight-hour increments over a 12-month period, the package is an affordable and flexible option for cloud customers to receive expert guidance on various aspects like onboarding, configuration, and data modeling. For more complex projects requiring detailed customization and architecture, Elastic also offers a Deployment Services package with longer consultant engagements.

Elastic Infrastructure 7.2.0 has been released, introducing significant enhancements to the application available on the Elasticsearch Service and included with the Elastic Stack. The most notable update is the addition of the Metrics Explorer within the Kibana Infrastructure app, which allows users to create time-series visualizations of their infrastructure metrics and refine these visualizations for custom dashboards. Additionally, the update includes layout improvements in the Inventory view to maximize space for metrics and enhance accessibility. This release also introduces beta modules for CoreDNS and Hashicorp's Consul, providing comprehensive metrics for monitoring Kubernetes, container deployments, and server cluster health. Users can access this new version by either upgrading their existing Elasticsearch Service cluster or downloading it as part of the Elastic Stack distribution.

The Elastic Stack 7.2 release introduces several significant updates aimed at enhancing observability, including new support for .NET applications in Elastic APM, an upgraded Metrics Explorer for better interaction with infrastructure metrics, and expanded Kubernetes monitoring capabilities with integrations for CoreDNS and CRI-O. The update emphasizes a unified approach to system health monitoring by consolidating disparate data sources into a single operational store using the Elastic Common Schema (ECS). This release also features improvements in Real User Monitoring (RUM) for single-page applications, agent-specific metrics collection, and enhancements in the Elastic Infrastructure app for metrics exploration. Elastic's commitment to supporting cloud-native technologies and Kubernetes is further demonstrated through new modules and integrations, alongside efforts to streamline log data management and active availability monitoring. These updates aim to provide a more comprehensive and efficient observability experience, fostering better operational insights and reduced administrative overhead for users, such as Red Wing Shoes, which utilizes Elastic's services to enhance visibility across its operations.

The release of Kibana 7.2.0 introduced a range of new features and improvements aimed at enhancing user experience and functionality. Key updates include the introduction of feature controls, allowing administrators to customize user access to various applications and tools within Kibana based on roles and privileges. The Saved Object Import and Export API now facilitates the seamless transfer of dashboards and their dependencies, while the Time Series Visual Builder supports rolled-up data visualization. Canvas enhancements include the ability to save custom elements and apply filters selectively, alongside a new autoplay feature. The release also marks the introduction of the Snapshot UI for managing Elasticsearch backups, and the integration of maps into dashboards, complete with dark mode and collapsible legends. Kibana's interface now supports Japanese, and the Machine Learning feature has expanded to include data frames and anomaly detection for sample datasets. Additional observability improvements include the Metrics Explorer and enhanced APM agent data collection. Users are encouraged to explore these new capabilities and provide feedback through the Kibana community forums.

Elastic Logs 7.2.0, a new release available on the Elasticsearch Service or as part of the Elastic Stack, introduces enhanced features for infrastructure and application log management. Key updates include support for structured logs with field pinning, enabling users to add custom columns in the log viewer, and a quick filter feature accessible directly from the log detail flyout. The release also expands Kubernetes and container monitoring capabilities with the addition of the NATS module, which captures and visualizes logs from the NATS messaging system, and the CoreDNS module, which supports various CoreDNS deployments and provides detailed query information. Furthermore, Filebeat's container input now supports Open Container Initiative-based Kubernetes CRI-O logs, allowing users to specify log formats, and the application can be accessed by creating or upgrading clusters on Elastic Cloud or through the default Elastic Stack distribution.

Elastic SIEM, now part of Elastic Security, was introduced as a beta feature in the 7.2 release of the Elastic Stack, offering a new approach to Security Information and Event Management (SIEM) through data integrations and a dedicated app in Kibana. This tool allows security teams to streamline host and network security workflows with features like the Timeline Event Viewer for investigation and evidence gathering. Elastic SIEM utilizes the Elastic Common Schema (ECS) to normalize data from various sources, facilitating cross-source correlation and analysis. With its free availability as part of the Elastic Stack's default distribution, Elastic SIEM has been adopted by organizations such as Bell Canada, Slack, Cisco Talos, and others for security analytics and threat hunting. It provides an interactive workspace for security practitioners in Kibana, enhancing capabilities for host and network security event analysis. The company plans to expand its offerings with features like detection rules and threat intelligence integration, as it continues to innovate and redefine traditional SIEM boundaries.

Beats 7.2.0 introduces enhanced processing capabilities and numerous new integrations, marking the latest stable release available for download. This version features a new script processor for edge event processing using JavaScript, which offers increased flexibility by allowing event manipulation without interacting with host systems. The release also expands Beats' integration options by adding eight new modules and a new Filebeat input, enhancing security analytics with modules for monitoring firewall logs from Palo Alto Networks and Cisco ASA, and introducing a NetFlow module. In the cloud-native domain, updates include a NATS module for Filebeat and CoreDNS modules for Filebeat and Metricbeat, along with a new container input in Filebeat for dynamic log collection, especially suited for CRI-O environments. Windows platform users benefit from new Sysmon and Security modules in Winlogbeat, as well as support for the Windows XML Event Log format. Users are encouraged to explore Beats 7.2.0 and provide feedback via Twitter or forums.

Elasticsearch 7.2.0, built on Lucene 8.0.0, introduces several enhancements aimed at improving relevance ranking, search efficiency, and system resiliency. Notable features include the ability to incorporate geo and time proximity into relevance rankings, enhancing the precision of search results, and a "typeahead search" feature that optimizes performance by quickly suggesting top-ranking results as users type queries. The release also improves resiliency with replicated closed indices and a new Snapshot Repositories UI in Kibana, which aids in data backup and management. Additionally, Elasticsearch SQL capabilities have been expanded to include geographical queries and case statements, and the new Data Frames plugin enables data transformation for deeper insights. Security enhancements include OpenID Connect support, providing seamless authentication across connected systems. These advancements aim to solidify Elasticsearch's role not only as a search engine but also as a reliable data repository and analytical tool.

Elastic App Search, previously available as a cloud-based solution, is now offered as a downloadable, self-managed search solution, allowing developers to deploy powerful and flexible search experiences on any infrastructure. This release marks a significant milestone as it expands the availability and flexibility of Elastic App Search, leveraging community feedback from its beta program. The solution integrates with Elastic Stack 7.2 or later, providing features such as simple data ingestion, powerful search APIs, UI frameworks, insightful analytics, and intuitive relevance controls, all designed to streamline the creation of engaging search experiences. Elastic App Search capitalizes on the robustness of Elasticsearch, offering the relevance, scale, and speed needed for diverse applications, while maintaining simplicity and ease of use, with the added benefit of being free to use alongside the default Elastic Stack distribution. Enhanced security features, including role-based access controls, are also available, ensuring secure and scalable deployments.

Elastic APM 7.2.0 introduces significant updates, including a beta .NET agent for ASP.NET Core and Entity Framework Core, enhanced Java application server support, and additional agent-specific metrics for improved observability and root cause analysis. The release also extends support to new Java application servers like Oracle's WebLogic and IBM's JBoss, as well as network frameworks such as OkHttp and JAX-WS client. Enhancements to the Real User Monitoring (RUM) agent now allow monitoring of Single Page Applications by capturing route change navigations. The update includes index lifecycle management for Elastic APM Server to automate index rollover while maintaining visibility. Users can now filter views based on deployment environments, and related errors are displayed on transaction sample views for easier navigation and error tracking. Elastic APM 7.2.0 is available on the Elasticsearch Service and as part of the Elastic Stack distribution.

Version 7.2 of the Elastic Stack brings significant enhancements across its suite of tools, notably introducing Elastic SIEM to bolster security analytics with expanded data collection and a dedicated interface for threat investigation. This release also marks the general availability of Elastic App Search for on-premises use, offering developers flexibility in deploying consumer-grade search experiences. The update advances observability with improved Elastic APM features, including a new Metrics Explorer for infrastructure insights and enhanced Kubernetes monitoring capabilities. Additionally, Elasticsearch, Kibana, Beats, and Logstash have received updates to improve user experience, processing efficiency, and system performance, reinforcing the Elastic Stack's position as a comprehensive solution for data management and analysis.

Logstash 7.2.0 has been released, offering significant updates and new features that enhance its usability and integration capabilities. The release introduces general availability for Java plugins, allowing developers to create Logstash plugins using Java without Ruby dependencies, which improves performance and widens the contributor base. The revamped JMS input plugin enhances integration with JMS technologies, offering better stability and TLS security, though it requires manual installation until the 7.3.0 release. Additionally, the update supports data output to the self-managed version of Elastic App Search, facilitating data migration for custom Elasticsearch application searches, with plans to bundle this in future releases. Enhancements to Google Cloud Platform integrations include improved reading and writing capabilities with Google Cloud Storage, thanks to a new Java client for better performance. Users are encouraged to test these new features and provide feedback through various channels.

Elastic Uptime Monitoring version 7.2.0 introduces significant enhancements, such as location support and integration with other observability solutions, available on the Elasticsearch Service or as part of the Elastic Stack distribution. This release enables users to monitor site availability from various geographical locations by adding location metadata to Heartbeat data, which can be visualized in Kibana's Uptime app. Additionally, it integrates Uptime with Logs, Infrastructure, and APM apps through bi-directional links, facilitating seamless navigation and enabling comprehensive observability for effective troubleshooting and root cause analysis. Users can access the updated Uptime application on Elastic Cloud or download it as part of the Elastic Stack to monitor their applications more effectively.

Kibana's recent updates have focused on enhancing visualization capabilities, improving user interface elements, and integrating geospatial data with Elasticsearch. Key developments include the merging of a PR to support Elasticsearch 7 in GDAL, facilitating the ingestion of shapefiles into Elasticsearch, and the expansion of elastic-charts with new features like custom tooltips and crosshair cursors. The Lens application is nearing the integration of new visualization types, including data tables and metrics, alongside configuration options for terms and date histograms. Canvas has introduced alignment and distribution options for layout precision, further enriched by keyboard shortcuts for element positioning. The ongoing efforts in application architecture include a comprehensive embeddables API and dashboard migrations, while design improvements in Elastic UI (EUI) focus on components for KQL auto-complete and CSS in JS. These updates collectively aim to streamline data visualization and manipulation for users, enhancing the overall functionality and user experience of Kibana.

Elastic has introduced an official Homebrew tap, allowing macOS users to more easily install and manage Elastic Stack software through Homebrew, a popular package manager in the developer community. This new tap provides access to both the Open Source Software (OSS) and default distributions of Elastic's offerings, including features like index lifecycle management, Elasticsearch SQL, and Canvas visualizations, with the added benefit of free security in the default distribution. The tap ensures that users have timely access to the latest software versions as updates are automated to coincide with Elastic's release process. The initiative is part of Elastic's effort to meet users where they are, as many of their engineers and users are already integrated into the Homebrew ecosystem. Additionally, through the Elastic Cares program, the company has made a donation to the Homebrew project to support its sustainability, highlighting Elastic's commitment to contributing to the open-source community.

Deploying Elasticsearch alongside relational databases like MySQL can enhance search capabilities, but it requires synchronization of data between systems. This blog post explains how to use Logstash with its JDBC input plugin to efficiently synchronize data from a MySQL database to Elasticsearch. The process involves configuring Logstash to periodically poll MySQL for records that have been inserted or updated since the last check, using fields like "modification_time" to track changes. By setting Elasticsearch's "_id" field to match the MySQL "id" field, direct mapping is achieved, and updates in MySQL overwrite corresponding documents in Elasticsearch efficiently. The approach resolves common synchronization issues by incorporating conditions like "modification_time < NOW()" in SQL queries to ensure each record is sent to Elasticsearch only once, avoiding data loss or redundancy. The synchronization setup requires specific configurations in MySQL and Logstash, and while it efficiently handles insertions and updates, deletion synchronization needs additional strategies such as soft deletes or dual-system commands. The outlined methods are tested with MySQL but are applicable to any RDBMS, offering a robust solution for maintaining consistency between MySQL and Elasticsearch.

Kibana's development efforts, as of June 17, 2019, focused on enhancing various features and migrating components to a new platform, with significant work on the security plugin's authentication and the Spaces plugin. The team aimed to ensure compliance with Content Security Policy by eliminating the use of `unsafe-eval`, while clustering improvements included mutual TLS certificates for security. Updates in Elastic Charts involve removing dependencies on EUI components and introducing new features like Histogram Mode, alongside various bug fixes. The Lens project achieved a milestone with field drag-and-drop functionality for chart creation, and ongoing work on data tables and dimension editors. Embeddables within Canvas saw progress with a proof of concept, and there were efforts to enhance documentation with more examples and function definitions. Meanwhile, the new embeddable API was integrated, although not yet widely implemented, and application architecture improvements continued with the transition away from Angular and the introduction of utilities for state containers. Updates to Kibana's EUI version brought new features and bug fixes, aiming for a closer sync between Kibana and EUI, while Elastic Charts and EUI established a more stable relationship to avoid dependency issues.

Leaseweb, a global Infrastructure-as-a-Service provider, transitioned from Splunk to the Elastic Stack to enhance security and observability by unifying data into a single platform, reducing costs, and improving administrative efficiency. The company, with over 80,000 servers across numerous data centers, faced challenges in monitoring and securing its vast infrastructure, leading to the adoption of Elastic for its open-source nature, responsive support, and machine learning capabilities. By integrating various data sources, such as firewalls and event logs, Leaseweb utilized Kibana dashboards to enhance proactive monitoring and incident response. Machine learning within Elastic enables the identification of suspicious activities and supports rapid incident mitigation, demonstrated by its ability to detect and respond to a DDOS attack. Leaseweb emphasizes the importance of selecting valuable data sources, staying updated with Elastic's releases, and leveraging support services to optimize their security infrastructure. The company plans to further develop behavior-based alerts, highlighting the proactive potential of machine learning in anticipating and addressing security threats.

Monitoring an Apache Kafka cluster can be efficiently managed using Filebeat and Metricbeat modules, which simplify the collection of logs and metrics by automating configuration and integrating seamlessly with Elasticsearch and Kibana. By setting up these modules, users benefit from standardization through the Elastic Common Schema, sensible index templates, and optimized shard sizes via the Rollover API. The example environment consists of a three-node Kafka cluster with each node running Kafka 2.1.1, along with Filebeat and Metricbeat configured via Cloud ID to send data to an Elasticsearch Service cluster. This setup enables detailed visualizations in Kibana, such as log throughput, consumer lag, and partition states, offering insights into the Kafka cluster's performance and consumer activity. The article encourages users to try this monitoring solution by offering a free trial of the Elasticsearch Service, highlighting the ease of setup and the comprehensive dashboards available for monitoring Kafka clusters.

Skopos Labs, a legal and financial data provider, has successfully integrated Elasticsearch and Elastic Cloud into its machine learning platform to analyze extensive datasets and forecast government policies' impacts on industries. Initially a startup with limited resources, Skopos Labs chose Elasticsearch for its developer-friendly, scalable full-text search capabilities, which facilitated handling semi-structured data and supported their rapid growth. Opting for Elasticsearch Service on Elastic Cloud over self-managed solutions on AWS or Google Cloud allowed the company to focus development efforts on product delivery without heavy DevOps investment. The service's features, including custom plugins and machine learning capabilities, have been crucial to their operations, as demonstrated when the snapshot feature rescued them from a data mishap. Over time, Skopos Labs has expanded Elasticsearch's use from a primary data store to roles in logging and anomaly detection, enhancing quality assurance. Engaging with the Elastic community, Skopos Labs remains confident in Elasticsearch's ability to support their evolving data needs and customer demands, as they plan for future expansions and features.

Zero Latency, a company focused on creating immersive free-roam multiplayer virtual reality experiences, utilizes the Elastic Stack to enhance both player safety and gameplay quality in their VR venues. Players, equipped with VR goggles and portable computers, freely navigate warehouse-scale spaces while game masters monitor their movements to prevent collisions. To address the challenges of managing numerous devices and ensuring player safety, Zero Latency employs Elasticsearch for real-time data analytics and infrastructure monitoring. By using Filebeat to collect structured JSON logs, they analyze player movements and proximity alerts to optimize space usage in games like Engineerium, a puzzle game with suspended plates. The Elastic Stack's flexibility allows Zero Latency to efficiently manage logs and analytics, enabling them to quickly identify and resolve hardware or software issues across their 30 global venues.

The text provides a detailed update on the development progress and upcoming features for Kibana as of June 10, 2019, highlighting several key areas such as platform migration, new visualization capabilities, design enhancements, and operational improvements. It notes the ongoing work on migrating various plugins to a new platform, including spaces and security, and mentions the introduction of new features like the ability to copy saved objects between spaces. The update also discusses enhancements to the Geo-Maps app and the integration of Elastic Charts, as well as the development of the Lens visual editor. Additionally, it touches on the transition to using Babel for compiling, improvements in data grids, and experimental work on compressed panels. The document concludes with an overview of efforts in alerting, reporting, and testing, underscoring a focus on refining Kibana's functionality and user experience.

Elastic has announced its acquisition agreement with Endgame, Inc., an endpoint security company, to merge their capabilities and enhance their security offerings. This strategic move aims to create a comprehensive security product by integrating Endgame's endpoint protection, detection, and response capabilities with Elastic's existing Security Information and Event Management (SIEM) features, leveraging Elasticsearch for enriched data analysis and visualization. The integration will provide a robust security solution, utilizing Elastic's search technology and Endgame's endpoint data collection capabilities, to help organizations prevent, detect, and respond to security threats more effectively. Both companies share a commitment to openness and user enablement, and they are aligned on product roadmaps and go-to-market strategies. The merger will allow Elastic to expand its SIEM functionalities and benefit from Endgame's expertise, while Endgame will gain access to Elastic's extensive user base and technological infrastructure. The transaction is subject to regulatory and shareholder approvals and is expected to enhance the security posture of clients by offering a unified and powerful security solution.

Event Query Language (EQL) is introduced as a powerful tool for enhancing threat detection and response, developed by Endgame to overcome the limitations of traditional indicator-based detection methods. EQL enables security practitioners to focus on adversarial behaviors using a syntax that is both accessible and robust, allowing users to express complex queries without needing deep technical expertise in database operations. The language supports real-time detection and hunting by facilitating intuitive and iterative data exploration, and it integrates seamlessly with Endgame's endpoint-focused architecture, ensuring efficient data processing and analysis without reliance on cloud connectivity. EQL supports sophisticated queries, including sequences of events and process ancestry, allowing for nuanced threat analysis and detection. By leveraging MITRE's ATT&CK framework, EQL advances the ability to detect unknown attacks and empowers defenders to swiftly respond to evolving threats. Through its design and application, EQL unifies search, hunt, and detection processes, ultimately improving the efficacy of security operations and contributing to a deeper collective understanding of security tools.

The Scottish Council for Voluntary Organisations (SCVO) has undertaken a significant technological transformation to improve its web services, primarily through the integration of Elasticsearch, which is used for both data storage and search capabilities. This initiative aimed to consolidate various standalone websites and databases into a unified platform, enhancing search accuracy and user engagement across SCVO's services, including Goodmoves and Volunteer Scotland Search. By leveraging Elasticsearch Service on Elastic Cloud, SCVO has effectively streamlined its development processes, allowing the team to focus on user experience and search query improvements rather than backend maintenance. This approach has resulted in increased site search usage and user satisfaction, highlighted by feedback praising new features like the Similar Jobs tab. Future plans include utilizing the join datatype to enhance data indexing efficiency and conducting a thorough test of their Disaster Recovery process to ensure robust system resilience. Calum MacÙisdean, SCVO's Web Development & Design Manager, has been instrumental in this digital transformation, drawing on his background in human-computer interaction and previous experience in tech startups and digital publishing.

With the release of Elastic Stack versions 6.8 and 7.1, Elasticsearch has made several security features freely available in the default distribution, focusing on encryption, authentication, and authorization. TLS encryption is now integrated into Elasticsearch and Kibana, eliminating the need for complex proxy setups to secure communications within clusters. Native authentication is also freely accessible, allowing for more streamlined user management without relying on potentially insecure proxy configurations. The authorization system has been enhanced to offer robust, flexible control over data access down to individual documents and fields, moving away from fragile, proxy-based solutions. Despite these advancements, traditional security measures like using VPNs, firewalls, and data backups remain essential, as they add additional layers of security. The introduction of new tools and services, such as the official Kubernetes Operator and enhanced scripting restrictions, aims to simplify securing Elastic Stack deployments. As the landscape of security constantly evolves, Elastic Stack continues to adapt by updating its practices and offering advanced security features to both cloud and on-premises users.