November 2018 Summaries

EQL, the Event Query Language developed by Endgame, is a versatile and extensible language designed to express relationships between security-relevant events, applicable to various use cases and independent of specific architectures. Originally part of Endgame's endpoint security product, EQL has now been released to the public, allowing broader use beyond Endgame's customer base. This release includes core language features, Sysmon schema mapping, and analytics focused on Atomic Blue, aimed at enhancing detection capabilities beyond traditional indicators of compromise (IOCs). EQL is praised for its minimal learning curve, intuitive search capabilities, and ability to work across different data sources without dependence on specific schemas, thus allowing for easy sharing of analytics across security teams. The release also includes EQLLib, a set of analytics to familiarize users with EQL, and Atomic Blue, which maps EQL logic to adversary techniques, contributing to the MITRE ATT&CK framework. Endgame plans to expand EQL's capabilities by supporting additional data sources and technologies, and seeks community feedback to improve and evolve the tool, recognizing its potential to address limitations in universally describing detection logic across datasets.

The blog post focuses on using Canvas, a feature within Kibana, to create a dynamic and visually customized data table, specifically an airport flight monitor. It highlights the flexibility of Canvas by demonstrating how to use built-in elements like the Data Table and Debug for data visualization and troubleshooting. The tutorial guides users through setting up the environment, utilizing Elasticsearch SQL for querying sample flight data, and employing functions such as `mapColumn`, `random`, and `formatdate` to manipulate data and format the table. It also covers debugging techniques, combining and excluding columns, and applying custom CSS for aesthetic enhancements. Readers are encouraged to explore Canvas's capabilities further by experimenting with additional elements and resources.

In Elastic's version 6.5 of their machine learning anomaly scoring, significant changes were introduced to improve the way anomalies are assessed and reported. The normalization of partitions now considers a maximum normalized score per partition, offering a broader range of scores and making it easier to identify significant anomalies across different partitions. Previously, normalized scores were influenced by more severe anomalies in other partitions, leading to lower scores for some significant events. Additionally, the introduction of multi-bucket analysis allows for the detection of anomalies that occur over multiple time buckets, which were previously difficult to identify. This change is highlighted by a new "cross" symbol in the user interface, indicating a multi-bucket anomaly, and the addition of a multi_bucket_impact field in the data. These enhancements are intended to provide more accurate and actionable anomaly detection and necessitate a review of alerting logic for users utilizing Elastic machine learning in their workflows.

Elastic App Search, built on top of Elasticsearch, was introduced as a self-managed beta release on November 28, 2018, allowing users to deploy it within their own infrastructure. This tool simplifies the creation of search experiences by providing an API wrapper for Elasticsearch that streamlines data indexing and search tuning, complemented by intuitive dashboard tools. Elastic App Search enables users to create and manage search engines, index documents, and perform queries, offering capabilities such as field weighting, result boosting, and real-time analytics. It supports a variety of data types and offers a user-friendly interface for configuring search relevance, viewing analytics, and managing search configurations. During the beta period, users can deploy it on their own hardware at no cost, provided they have the necessary dependencies, including Java and a compatible version of Elasticsearch.

The Kibana team has been actively working on various projects and updates, including progress on granular application privileges, enhancing alerting services, and integrating K7 design elements like the ML conversion to SASS and filter bar updates to EUI. Efforts also continue on the Geo GIS app and Canvas integration, with new features such as URL parameter support and persistent grouping of elements. A new visualization editor is being developed, incorporating field statistics for enhanced data insights. The design team is advancing the SASS conversion and finalizing visual patterns for the global date picker, while TypeScript improvements and bug fixes are ongoing across different components. Operational updates include running license checks, removing Husky, and fixing non-conforming licenses. The KibanaApp team is focusing on visualizations, bug fixes, localization, and EUIfication of the filter bar, with several pull requests addressing improvements and bug resolutions. Overall, the team is preparing for upcoming releases, including version 6.5.1 and anticipating future enhancements.

Elastic has announced the general availability of its Elastic APM Java Agent, following a successful beta phase. This Java Agent, requiring APM Server 6.5, is part of Elastic's Application Performance Monitoring solutions, which provide insights into application performance and error tracking. The agent offers features such as automatic instrumentation for supported frameworks like Spring MVC/Spring Boot, visibility into web requests, data store interactions, and networking, as well as distributed tracing capabilities for multi-service environments. Users can customize their monitoring using the provided API, enabling tracing of specific methods and error tracking. Additionally, the agent supports integration with the OpenTracing API, allowing for seamless combination of automatic and manual instrumentation. Elastic plans to enhance future integrations with reactive frameworks and improve the integration of logs, metrics, and traces. Feedback from users is encouraged through a survey and the Discuss forum.

The article introduces Heartbeat, a tool designed for uptime monitoring within the Elastic Stack, which became generally available with the 6.5 release. Initially a beta release in January 2017, Heartbeat acts as a lightweight shipper that monitors network endpoints using HTTP, TCP, or ICMP, and reports the status to Elasticsearch. Configuration involves listing URLs to monitor in a heartbeat.yml file, with periodic checks being executed to verify endpoint status. Data collected by Heartbeat is visualized in Kibana through pre-built dashboards, allowing users to detect issues such as outages or performance degradations. The tool can be combined with other Elastic Stack components like Metricbeat and APM for a comprehensive operational overview, aiding in faster incident resolution. Users can also customize dashboards or create new visualizations based on Heartbeat data, and leverage alerting and machine learning features for enhanced monitoring. The article encourages users to explore Heartbeat's functionality through its documentation and community forums.

Elastic App Search Reference UI is an open-source developer tool integrated into the App Search dashboard, designed to facilitate the creation and testing of robust search experiences. Written in React, it provides a dynamic search preview where developers can experiment with features like facets, pagination, and sorting to improve search relevance, which is crucial for fulfilling user and business needs. The tool supports developers by generating a quality design and developmental foundation for new search interfaces, allowing them to configure fields such as title, facet, sort, and URL fields for customized search experiences. Users can further refine their search results using interactive tools like faceting and sorting, enhancing the overall search experience. The Reference UI can be downloaded and customized, enabling developers to run a local version that is ready to fork and build upon, with the potential to inspire innovative search solutions within the community.

The Elastic APM Go Agent 1.0.0 has been officially released, marking its general availability with enhanced features, bug fixes, and optimizations for monitoring Go applications. As part of the Elastic Application Performance Monitoring (APM) suite, which supports several programming languages including Java, Node.js, and Python, the Go agent allows developers to instrument their applications by fetching the agent source and using built-in modules or custom spans to monitor performance and capture errors. This release introduces changes such as support for distributed tracing in Elastic Stack version 6.5.0 or greater, a new V2 intake protocol for more efficient event streaming, and improved OpenTracing integration for better correlation of spans with transactions. Additionally, new instrumentation modules for Redigo and go-restful frameworks have been implemented, and the agent's import path has been updated to "go.elastic.co/apm". Developers can visualize application performance data in the APM UI, with insights into request numbers, response times, and code path contributions to performance, while also participating in community discussions and surveys to influence future updates.

Kibana Spaces, introduced in version 6.5, allow users to organize dashboards, visualizations, and other saved objects into distinct categories, enabling better management and access control. This feature is particularly beneficial when using security protocols, as it allows administrators to determine which roles have access to specific Spaces within the same Kibana instance. Existing saved objects are initially placed in the "Default" space during the upgrade, and users can move objects between Spaces using the Import/Export UI or APIs. While the Import/Export UI is suitable for individual objects, moving entire dashboards with dependencies requires using the Dashboard Import/Export APIs. This process involves exporting the dashboard with its dependencies to a JSON file and importing it into the desired Space, ensuring that each Space's saved objects remain separate. This functionality also facilitates the consolidation of multiple tenants into a single Kibana instance by using Spaces to maintain isolation and organization, ultimately enhancing user experience and management efficiency.

Kibana Spaces, introduced in version 6.5, is a feature designed to help users organize their dashboards, visualizations, and other saved objects into distinct categories, each operating independently to prevent clutter across spaces. Users can create, edit, and delete spaces via the Spaces management UI, where they can assign a unique URL identifier for each space. Security features allow the customization of access levels through role-based permissions, with three levels of minimum privilege: all, read, and none, providing flexibility in managing access to different spaces. The system facilitates the movement of objects between spaces via import/export and offers the capability to view a summary of space privileges for each role. Additionally, starting with Elastic Stack versions 6.8 and 7.1, role-based access control is available for free in the default distribution, enhancing the overall security management of spaces without the need for a Gold or Platinum license.

In the November 2018 update on Kibana, significant progress was reported across multiple areas, including granular application privileges, accessibility compliance, and GIS app styling enhancements. The team focused on improving accessibility by performing audits of version 6.5, while also enhancing the GIS app with static and data-driven styling capabilities. Developments in Elastic Map Service involved the migration of Sophox to the production system to enhance reliability. Canvas integration saw bug fixes and the planning of GA features, with creative projects like a workpad displaying planes over Phoenix, showcasing the platform's versatility. Index Lifecycle Management and Cross-cluster Replication features were advanced, with initial PRs enabling CRUD app development and replication relationship management across remote clusters. The design team continued transitioning to Sass, implementing a Sass linter, and moving towards TypeScript support, among other UI and UX improvements. Additionally, various bug fixes and enhancements were made in the KibanaApp, including updates for React compatibility and translation improvements, all while QA testing for version 6.5.0 was ongoing.

Beats 6.5.0 marks a significant update in the Beats project, introducing various new features, improvements, and modules, many of which are in beta. Central management allows users to control Filebeat and Metricbeat configurations through Kibana, simplifying the management of extensive deployments. The release also debuts two new Beats: Functionbeat, designed to run serverless on AWS Lambda for log transfers, and Journalbeat, aimed at ingesting logs from journald on Linux systems. Additionally, the Infrastructure and Logs UIs provide enhanced visualization and navigation for Metricbeat and Filebeat data within Kibana. Heartbeat achieves general availability, bringing dynamic service monitoring capabilities, while the Suricata integration via a Filebeat module eases the collection of intrusion detection logs. Improvements across existing modules include Kafka support in Metricbeat and Filebeat, expanded metrics in Redis and Memcached modules, and new metricsets in MongoDB and PHP FPM modules.

Elasticsearch 6.5.0, based on Lucene 7.5.0, introduces several key features and improvements across various areas, including cross-cluster replication, minimal snapshots, SQL/ODBC enhancements, search use cases, JDK 11 support, security, and machine learning. The new cross-cluster replication feature allows data replication across clusters, enhancing multi-datacenter systems' efficiency. Minimal snapshots reduce storage space by up to 50% by storing only essential information for recovery. The release also includes an initial ODBC driver for Windows, expanding SQL capabilities and integration with BI tools. Enhanced search functionalities introduce scriptable token filters and a new annotated_text type for NLP applications. Elasticsearch now supports JDK 11 and the G1 garbage collector, addressing previous issues with large heaps. Security improvements feature authorization realms and a new structured audit logging format, while machine learning updates refine anomaly scoring and introduce multi-bucket analysis, alongside an experimental find file structure API for better data ingestion.

Kibana 6.5.0 introduces several high-impact features, including Spaces, Rollup support, and Canvas, alongside various medium and smaller improvements such as enhanced Console functionalities. Spaces allow users to organize their saved objects into meaningful categories, and Canvas offers a platform for creating dynamic data presentations with rich styling capabilities. The release also includes a Management UI for creating and managing rollup jobs, enabling compact data storage for historical analysis, and beta features for visualizing rolled-up data. Additionally, the update brings two new sample data sets targeting business analytics and website traffic analysis, and introduces the File Data Visualizer for uploading and analyzing log files, which is currently experimental. The release also debuts Infrastructure and Logs applications, enhancing operational observability through logs, metrics, and application traces, and expands the Console's auto-complete capabilities and documentation access.

webLyzard technology, in collaboration with the United Nations Environment Programme (UNEP), has developed a semantic search platform using Elasticsearch to track global communication flows and visualize public debates on environmental issues related to the UN's Sustainable Development Goals (SDGs). This platform, aimed at stakeholders such as policymakers, corporations, and NGOs, extracts knowledge from diverse online sources to enhance understanding of complex issues, track debate evolution, and identify public opinion indicators. The visual analytics dashboard, powered by Elasticsearch, includes various interactive tools for analyzing public discourse by topic and location. The platform's cross-lingual capabilities allow consistent macro-level views of SDGs across regions, aiding NGOs and citizens in tracking international debates. Migration to Elasticsearch 6.4.2 has significantly improved the system's performance, with response times nearly halved for complex queries, enabling future research and collaborations with stakeholders and organizations supported by EU and other programs.

Elastic identified a vulnerability in Kibana's PDF reporting feature, which inadvertently transmitted user authentication credentials in HTTP headers when requesting data from external resources. The vulnerability, affecting versions 4.0 to 4.6, 5.0 to 5.6.12, and 6.0 to 6.4.2 of the Elastic Stack, exposes credentials to external resource providers, although there is no evidence of unauthorized use. Users employing this feature should change their credentials as a precaution. Elastic released fixes in versions 5.6.13 and 6.4.3, and provided guidance for managing credentials across different authentication realms, such as Native, File, LDAP, Active Directory, and SAML. Users unable to upgrade are advised to follow recommended security practices and reach out to support if needed.

GoDaddy, renowned for web hosting and domain management, has leveraged Elastic machine learning to enhance user experience by efficiently managing its vast data infrastructure. With over 17 million customers and 75 million domains, GoDaddy handles more than 200,000 messages per second, including DNS queries and system logs, using the Elastic Stack. Initially, disparate Elasticsearch clusters were set up by different teams, leading to data silos, but in 2014, a dedicated team was formed to streamline Elasticsearch deployment, managing over 60 clusters with 270 TB of data from an 11 PB HDFS environment. These efforts enabled GoDaddy to improve patch compliance across its servers and provide detailed insights into system performance and user experience through centralized logging and machine learning for anomaly detection. By collaborating with Elastic experts, GoDaddy has successfully implemented machine learning to track user interactions and detect anomalies, enhancing site performance and enabling data-driven monitoring of business KPIs.

Elasticsearch Service enhances its security features by integrating Security Assertion Markup Language (SAML) and multi-factor authentication to provide robust identity management and secure access. SAML, now supported from version 6.4 and above, allows for seamless integration with external identity providers, simplifying user management and enabling a single sign-on experience across applications. This feature eases administrative tasks like employee onboarding and enhances security by delegating authentication responsibilities. Additionally, Elasticsearch Service now supports multi-factor authentication, offering options such as Google Authenticator or SMS as a secondary authentication layer, which addresses frequent security breaches involving user credentials by requiring short-lived tokens. These enhancements are available to all Elastic Cloud subscriptions, and users can explore these features with a 14-day free trial of the Elasticsearch Service.

Kibana's Canvas feature offers users a versatile platform for creating dynamic and visually appealing data presentations using built-in elements like Metrics and Markdown. The blog post provides a detailed tutorial on leveraging these elements with sample flight data available in Kibana versions 6.4 or greater. It guides users through setting up workpads, installing sample data, and utilizing SQL queries to manipulate and display data. The tutorial demonstrates how to create Metric elements to show total and average flight delays, incorporating time filters and customization options to refine the presentation. Similarly, it explains how to use Markdown elements to display the total number of flights and categorize flight delay types, emphasizing the application of functions like `filterrows` to exclude irrelevant data. The post encourages experimentation with Canvas's full capabilities and provides links to additional resources for further exploration.

Elastic App Search introduces a new feature that allows users to explore its functionalities using a pre-loaded National Parks demo engine, providing a hands-on experience without the need to bring their own data. The demo contains data on 59 U.S. National Parks, enabling users to experiment with features such as Relevance Tuning, Proximity Boosts, and Curations to enhance search results. The platform's dashboard allows real-time adjustments to search queries, improving user experience and trust by promoting relevant results like the Rocky Mountain National Park for mountain-related searches. Users can track analytic events to optimize search performance further, ensuring their search experience aligns with user needs. The National Parks dataset is licensed under Creative Commons and authored by Kevin Nayar, and users can explore Elastic App Search with a 14-day free trial.