February 2020 Summaries

In a recent update, Elasticsearch introduced several notable improvements and changes to enhance user experience and address existing issues. These updates include better query cancellation, which now propagates cancellation to all nodes and child tasks to prevent unnecessary CPU and I/O usage, although some operations like terms dictionary expansion and complex BKD reads still need more efficient handling. Mapping validation has been improved to catch misconfigurations earlier in the process, especially during index creation, providing users with a more seamless experience. The migration from Joda to Java time revealed the need for better backward compatibility, prompting Elasticsearch to restore support for pre-7.0 indices and improve migration documentation. Lucene's integration now includes the updated Snowball stemmer, offering support for additional languages such as Hindi and Tamil. Integrity checks via Lucene's CheckIndex tool are being enhanced to catch bugs earlier, and a proposal for a more flexible Korean dictionary mechanism is under consideration. The update also introduces breaking changes like disallowing duplicate percentile values in percentile aggregations and boosting the default scroll size in the Elasticsearch Hadoop Plugin.

Logstash, an open-source data processing pipeline, can be optimized for maintainability and reusability by using modular components stored in separate files, which are combined to create pipelines. This approach addresses the complexities and code duplication issues that arise when processing events from multiple input sources. By utilizing the glob expression to specify which files to include, Logstash can maintain an alphabetical order of execution, ensuring that shared logic across pipelines is effectively managed. The blog illustrates the setup of unique pipelines using components stored in specifically named configuration files for inputs, filters, and outputs, demonstrating how this modular technique simplifies pipeline management and enhances code clarity. Furthermore, this method allows for the reuse of common logic across different pipelines, with the potential addition of pipeline-to-pipeline communication to further enhance modularity.

Resource-based pricing in security, as advocated by Elastic, represents an innovative approach that addresses the challenges associated with traditional per-endpoint and ingest-based pricing models. These conventional methods often lead to overpayment and inefficiencies, as organizations must predict future needs and pay upfront, potentially resulting in unused capacity or limited data analysis due to cost constraints. Elastic's model charges customers based on data usage, specifically compute, storage, and memory, rather than fixed metrics like endpoints or data ingested, allowing for greater flexibility and cost-effectiveness. This approach enables organizations to deploy security solutions at their own pace, make informed decisions about data retention and query speeds, and avoid surprise costs through consumption capping. Elastic's resource-based pricing facilitates seamless integration across different use cases, such as security, observability, and analytics, without the need for duplicative infrastructure or extensive retraining, thereby enhancing both security outcomes and operational predictability.

Elasticsearch Service on Elastic Cloud offers a comprehensive and cost-effective solution for managing data storage as workloads expand, particularly in observability and security domains. It provides strategies like hot-warm deployments and index lifecycle management (ILM) to optimize storage by utilizing hardware profiles that separate frequently and infrequently accessed data, significantly reducing costs while maintaining data accessibility. By implementing ILM, users can automate data movement between hot nodes with faster storage and warm nodes with higher storage density. Additionally, data rollups offer further savings by summarizing data into less granular documents, which frees up storage space and reduces costs. These methods allow users to manage data efficiently, extending the retention period or reducing cluster size without sacrificing performance or visibility. Elastic Cloud's robust capabilities are supported by operational expertise from the creators, and users can explore these strategies through a free 14-day trial.

As attackers increasingly use "living off the land" techniques to bypass modern security software, researchers have developed a graph-based framework called ProblemChild to detect anomalous parent-child process relationships. This approach utilizes machine learning to generate weighted graphs from process creation events, allowing for the identification of suspicious activity amid the normal noise of system operations. By leveraging community detection and a prevalence service, the method distinguishes between common and rare process chains, reducing false positives and enhancing detection accuracy. The framework, demonstrated at VirusBulletin and CAMLIS, aims to simplify the detector writing process, requiring less domain expertise and enabling more effective identification of attack sequences. Additionally, the prevalence engine helps security professionals understand the rarity of relationships between events, offering a new perspective on threat detection.

Elastic Security showcased its comprehensive, unified threat protection and security analytics solution at the RSA Conference 2020, emphasizing its recent advancements in version 7.6.0. The solution integrates Elastic Endpoint Security and Elastic SIEM to provide real-time threat detection and response capabilities across various operating systems, leveraging open-source Elasticsearch. With a new SIEM detection engine that aligns with the MITRE ATT&CK™ framework, Elastic Security automates threat detection and offers prebuilt rules to identify both known and unknown threats, aiming to reduce the mean time to detect to zero. The platform provides deep visibility, surpassing even Microsoft's capabilities, to detect malicious activities and enables automated responses for threat remediation. Elastic Security empowers security teams by correlating data from multiple sources, offering real-time analysis and visualization to streamline security operations, and enhancing workflow automation to allow analysts to focus on critical investigations and problem-solving.

Elasticsearch, an open-source search and analytics engine developed by Elastic, is widely used for its speed, scalability, and ability to handle various data types. However, its popularity and open-source nature also pose security risks, as unsecured clusters left accessible on the internet can lead to data breaches. Elastic provides default security features to mitigate such risks, including authentication and TLS encryption, which are essential for protecting Elasticsearch clusters from unauthorized access and eavesdropping. Users can determine if their clusters are secure by querying the settings API, and Elastic offers resources and community support to help users implement robust security measures. Elastic Cloud, a managed service by Elastic, automatically includes security configurations, underscoring the importance of proactive security management in safeguarding sensitive data stored in Elasticsearch.

Elastic has integrated its own security solutions, Elastic Endpoint Security and Elastic SIEM, into its internal operations to enhance threat detection and response capabilities, achieving unprecedented visibility and efficiency. By partnering with Endgame and Perched, Elastic expanded its security expertise and incorporated advanced features such as automated responses and comprehensive analytics into its Elastic Stack platform. The company initially piloted the migration from CrowdStrike to Elastic Endpoint Security, successfully deploying it across its endpoint fleet with minimal issues, and achieved a low false positive rate in threat detection. The seamless integration allowed Elastic to remotely address security threats across its global workforce, exemplifying enhanced collaboration and efficiency. Elastic's commitment to being "Customer Zero" ensures its solutions are rigorously tested internally before being offered to customers, showcasing a robust internal partnership and dedication to delivering superior security products.

In the final installment of a series on Mac system extensions for threat detection, the shift from kernel extensions to Apple's new EndpointSecurity and SystemExtensions frameworks is explored. Announced at WWDC 2019, these frameworks will replace kernel extensions in macOS 10.16, as third-party vendors will no longer be permitted to run in the kernel. The EndpointSecurity framework offers a userland alternative to the in-kernel Kauth and MAC frameworks, providing detailed system event information and allowing developers to more easily debug their applications. SystemExtensions, which run in user mode, mitigate the risks associated with kernel code development by enabling modern debugging tools and preventing system crashes from faulty code. These frameworks are tailored to enhance security applications by allowing event subscription, authorization, and notification handling through userland APIs, thus improving development efficiency and system stability.

Elastic has made significant advancements in enhancing the accessibility of its products, particularly in Kibana, by focusing on both mindset and tools to create inclusive user experiences. The company has introduced an accessibility statement and made numerous improvements in its Elastic UI (EUI) component library, ensuring consistent and accessible experiences across various interfaces. A key highlight is the integration of automated accessibility testing using axe, which, alongside training for QA and design teams, helps identify and address accessibility issues early in the development process. Additionally, Elastic is working on improving navigation for screen reader users, especially in single-page applications, and is committed to ongoing enhancements, including in its charting library. The company encourages community involvement in identifying and addressing accessibility challenges to foster a more inclusive environment for all users.

Elasticsearch has introduced improvements to enhance feature parity between Prefix Trees and BKD-backed geo_shapes, addressing limitations in multi-point and distance queries, which are vital for geospatial applications. A recent PR in Elasticsearch now allows multi-point queries with BKD-backed geo_shapes, enabling users to query multiple discrete points like cafes in a city or flu outbreaks across countries. Similarly, a PR in Lucene has added distance query capabilities to LatLonShape and XYShape, expanding the potential to perform common operations like identifying all values within a specified radius. Additionally, Elasticsearch has unveiled the Painless Lab, a new tool for experimenting with Painless scripts, offering immediate feedback, improved user interface, and extensive documentation links to enhance user proficiency in areas like Watcher and ingest node pipelines. The update also includes enhancements in S3 repository compatibility, providing options for region and signer algorithm overrides to support a broader range of third-party S3 implementations, such as Ceph and Minio configurations, although full support is not guaranteed.

Elasticsearch Service is now available in the Azure Tokyo region, marking the fifth global Azure region and the second in Asia for Elastic Cloud. This expansion allows existing and new users to access features like security, index lifecycle management, and machine learning, alongside intuitive tools such as Kibana Lens. The service supports various use cases, including app search, logging, and business analytics, through easy-to-use deployment templates. The move aligns with Japan's IT modernization efforts towards cloud-based solutions, providing a SaaS-first option for enterprises and SMBs. Elastic plans further Azure region expansions to enhance the Elastic Stack experience, and users will soon be able to purchase Elasticsearch Service subscriptions via the Azure Marketplace.

Kibana's latest updates as of February 2020 highlight improvements in accessibility, security, and platform functionality, aligning with the recent 7.6 release. The team introduced an accessibility statement to demonstrate their commitment to building inclusive user interfaces, while security enhancements include the deprecation of the kibana_user and kibana_dashboard_only_user roles in favor of more specific Kibana Privileges. Additionally, the platform now supports CSS and Sass in plugins through webpack imports, and the validation library @kbn/config-schema can parse arrays and objects from JSON strings, facilitating automatic parsing of various JSON structures. The completion of the Kibana Platform Testing Guide offers examples and tips for testing plugins effectively across unit, integration, and end-to-end scenarios, supporting developers in optimizing their workflows.

On Valentine's Day, the Elastic Community Team expressed their heartfelt gratitude to user group organizers for their dedication and passion in supporting the Elastic Community's growth and success. This appreciation message was directed to a vast network of organizers across various regions, including EMEA, APJ, AMER (West and South Central), AMER (East and North Central), AMER (Canada), and AMER (Latin America). The note highlighted the collaborative efforts of numerous individuals who have significantly contributed to nurturing local user groups, thereby enhancing the community's global presence and connectivity.

Centralized logging is crucial for gaining visibility into complex software systems composed of multiple moving parts, and timestamps play a pivotal role in this process by defining the order of events and aiding in the investigation of causality. The choice of timestamp representation, whether numeric or string-based, impacts several aspects, such as precision and the range of values. Numeric representations, like POSIX time, are efficient and easy to parse, while string representations, such as those specified by ISO 8601 and RFC 3339, offer flexibility and human readability. Best practices for handling timestamps in log aggregation include maintaining explicit time zone information, ensuring sufficient precision, and synchronizing clocks across systems. Tools like Elastic Observability, Filebeat, and Logstash facilitate parsing and forwarding log events, making it easier to correlate data from various sources. By understanding the complexities of timestamps and leveraging the right tools, organizations can effectively manage centralized logging systems.

In partnership with Defending Digital Campaigns (DDC), Elastic announced the provision of free, monitored Elastic Endpoint Security to 2020 U.S. presidential and congressional campaigns to enhance their cybersecurity defenses. DDC is a non-partisan organization that offers affordable security products and services to federal campaigns, addressing the significant hurdles of high costs and the need for expertise in forming effective security strategies. This collaboration, supported by the U.S. Federal Election Commission, leverages Elastic's security technology and expertise to help campaigns protect themselves against cyber threats and election interference. Michael Kaiser, President and CEO of DDC, highlighted the critical importance of bolstering the cybersecurity of political campaigns to safeguard democracy. Elastic Security, having previously assisted in the early stages of the 2020 election, aims to expand its impact through this partnership, ensuring the integrity of the electoral process.

Elasticsearch 7.6 introduces language identification to improve search relevance in multilingual corpora, utilizing language-specific analyzers for better tokenization, stemming, and decompounding. Documents are indexed using strategies like language per-field or language per-index, each offering unique benefits and challenges. The language per-field strategy allows for a single index with multiple language fields, facilitating language-specific analysis and boosting, while the language per-index strategy creates separate indices for each language, simplifying queries and scaling. Language identification enhances search accuracy by ensuring terms are analyzed appropriately, but its application to query strings is limited due to their brevity. The choice between indexing strategies depends on factors like management complexity and performance needs. Ultimately, language identification in Elasticsearch enables more precise and relevant multilingual searches, supporting diverse languages and scripts across different contexts.

Elastic Metrics 7.6.0 introduces a range of enhancements for monitoring cloud services, including new integrations for Google Cloud Platform (GCP) and Azure Storage, alongside expanded capabilities for AWS billing and usage tracking. The release improves observability by incorporating new metricsets for AWS services, such as billing and usage, allowing for efficient resource management and cost tracking. It also provides custom inventory views for AWS resources like EC2 instances and S3 buckets. The GCP integration leverages Stackdriver for comprehensive metric collection, while Azure Storage monitoring is enhanced with a new metricset. Additional improvements include expanded Kubernetes support with metrics for PersistentVolumes, PersistentVolumeClaims, and Services, as well as updates to the system module in Metricbeat to deliver detailed server behavior insights. The release also introduces new metricsets for Apache ActiveMQ, Kafka, and NATS Streaming services, further bolstering the platform's ability to provide detailed insights across various infrastructures.

Elastic APM 7.6.0 introduces significant enhancements, including support for messaging frameworks like Kafka and JMS, and the ability to act as a Jaeger intake, allowing seamless integration of Jaeger-instrumented traces into Elasticsearch without modifying existing code. The release also includes deployment annotations, providing automatic visualization of deployment impacts on application performance, and improved integration with the SIEM app to enhance security threat detection using APM data. Additionally, API key authentication is now available, offering more flexibility in managing services through multiple tokens and refined authorization rules. The update further enhances customization capabilities by supporting the configuration of APM indices from Kibana, facilitating a more tailored monitoring environment. Users can access Elastic APM 7.6.0 via the Elasticsearch Service on Elastic Cloud or download it as part of the Elastic Stack distribution.

Elastic Enterprise Search has introduced updates with the release of Elastic App Search 7.6 and Elastic Workplace Search 7.6, enhancing scalability and user experience across different platforms. The rebranding under the "Elastic Enterprise Search" umbrella reflects the company's focus on delivering enterprise-ready search solutions that are capable of supporting diverse users with speed and relevance. A significant feature in this release is the introduction of meta engines, which allow users to search across multiple engines, providing scalability and control over search operations without the need to index content multiple times. This update also marks the launch of App Search on Elastic Cloud, simplifying deployment across 24 regions through global cloud partners like GCP, AWS, and Azure. Elastic Workplace Search, still in its beta phase, now supports the SAML protocol for improved security, allowing organizations utilizing single sign-on to manage access more efficiently. Existing Elastic Cloud customers can immediately access these updates, while new users are encouraged to start with a 14-day free trial.

Elasticsearch 7.6.0, based on Lucene 8.4.0, introduces numerous enhancements and new features aimed at improving performance, cluster management, and machine learning capabilities. This release includes significant optimizations for queries sorting by dates and numbers, as well as improvements to geospatial queries and composite aggregations. The introduction of a new histogram data type and string stats aggregation enhances data handling capabilities. Machine learning receives a boost with the end-to-end pipeline now allowing for labeled data use, supporting applications like language identification. Cluster management is streamlined with better integration between index and snapshot lifecycle management, along with improved TLS diagnostics and simplified cross-cluster operations via proxy mode. Elasticsearch 7.6.0 also brings a visual mapping editor for easier index template management and promotes vector distance functions to general availability, demonstrating an ongoing commitment to enhance search capabilities and user experience across various domains.

Elastic Observability 7.6 introduces significant enhancements, including expanded multicloud monitoring support for Google Cloud Platform, native support for Jaeger traces, and machine learning-assisted log categorization. The release strengthens cloud monitoring by offering new modules for GCP, AWS Inventory views, and Azure Storage monitoring, while doubling down on open standards by integrating with Jaeger in Elastic APM. The update also simplifies root cause analysis through log categorization and streamlines cost management with AWS billing and usage analysis. Furthermore, the Elastic Uptime feature now includes a global visualization map, and a new integration between APM and SIEM enhances data sharing for security threat detection. These advancements aim to enhance visibility, efficiency, and collaboration across cloud operations and security monitoring.

Elastic Logs 7.6.0 introduces significant enhancements, including log categorization for improved anomaly detection, support for ingesting logs from Dockerized applications, AWS VPC flow logs, and ActiveMQ. Log categorization groups similar log messages into categories, allowing machine learning algorithms to identify anomalies by comparing current data against historical models without requiring predefined log patterns or deep technical knowledge. This feature simplifies anomaly detection by automating the identification of unexpected behaviors in log data, enhancing troubleshooting and recovery processes. Additional updates include the expansion of log sources, beta support for AWS Elastic Load Balancer, and an experimental Elastic Logging Plugin for Docker, which streamlines logging from Docker apps to Elasticsearch. The release also introduces Functionbeat support for Google Cloud, enabling log ingestion from Google PubSub and Google Cloud Storage. These enhancements aim to improve user experience and expand the capabilities of log management and analysis within the Elastic ecosystem.

The latest updates in Kibana, as of February 11, 2020, include new opportunities for employment, such as roles for a Senior Product Designer, a Senior JavaScript Engineer, and a Principal Product Manager among others. Developers can now utilize the new getServerInfo() API to access Kibana’s public URL, and benefit from enhancements in the app architecture with the SearchBar and TopNavMenu components, which now come with pre-wired default behaviors for managing filter and query states, auto-refresh settings, and saved query workflows. These components can be activated by setting useDefaultBehaviors to "true," though observables and callbacks are still required for notifications on state changes. Additionally, the updated Plugin Generator tool is available for creating new Kibana Platform plugins, aligning with the latest plugin structure and guidelines, and feedback is encouraged for further improvements.

Elastic Uptime Monitoring 7.6.0 introduces enhanced features for users of the Elasticsearch Service and the Elastic Stack, including a new visualization method for Heartbeat monitors in various geographic locations. The release offers a world map feature for an immediate overview of monitor statuses, eliminating the need to examine individual test results, and an expanded monitor list view that shows the most recent status by location. Users can now filter test results by tags, beneficial for managing large Heartbeat deployments, allowing focused monitoring. The updated version is accessible on the Elasticsearch Service on Elastic Cloud and can be downloaded as part of the Elastic Stack, facilitating streamlined monitoring of applications and services.

The guide outlines the recommended procedures for upgrading Elastic App Search to ensure access to new features, security enhancements, and performance improvements while minimizing disruptions in production environments. It emphasizes the importance of testing upgrades in a development environment and backing up data using Elasticsearch snapshots to safeguard against data loss. The document describes three primary upgrade methods: simple in-place upgrades with downtime, snapshot-based upgrades without downtime but requiring client-side changes, and in-place upgrades in read-only mode for App Search version 7.6 and later, which do not require new infrastructure or client configuration changes but make rollback more challenging. The choice of method depends on factors such as deployment mode, resource availability, and tolerance for service downtime. For Elastic Cloud users, the guide suggests either a default in-place upgrade with downtime or a snapshot-based process for uninterrupted service. Additionally, it highlights the importance of troubleshooting potential upgrade failures by checking application logs and suggests filing a support case with Elastic if issues persist.

Elastic Stack 7.6.0 introduces significant enhancements across its suite of products, focusing on improved performance, automated threat detection, and expanded machine learning capabilities. The release includes a faster Elasticsearch with block-max WAND optimization for sorted queries, making it more efficient for observability and security tasks. Elastic Security now features a new SIEM detection engine with nearly 100 ATT&CK-aligned rules, offering automated threat detection and improved visibility into Windows systems. The update also enhances supervised machine learning, allowing end-to-end capabilities from model training to inference within the stack. Elastic Enterprise Search, now branded as Elastic Workplace Search, introduces meta engines for unified search across multiple platforms, while Elastic Observability expands cloud integrations with AWS and GCP for better monitoring and introduces native Jaeger support for request tracing. These updates aim to make data operations more efficient and accessible, enhancing user experience across observability, security, and enterprise search domains.

Logstash 7.6.0 has been released, featuring the new JDBC Integration Plugin that combines input/output/filter plugins for JDBC into one cohesive unit, thereby streamlining plugin management and ensuring consistent configurations. The release also introduces a unified deprecation logging system, enabling users to track features that may become obsolete in upcoming major upgrades. Moreover, monitoring Logstash performance with Elastic Cloud has been simplified, allowing users to easily configure monitoring settings using the Cluster ID from the Elastic Cloud Admin Console. Users are encouraged to download the new version, engage with the community on social media or forums, and report any issues on GitHub.

Elastic Security 7.6 enhances its capabilities by integrating Elastic Endpoint Security and Elastic SIEM into a unified interface to provide comprehensive threat protection and visibility. This release introduces a new SIEM detection engine with nearly 100 out-of-the-box ATT&CK-aligned rules, which automate the detection of threats, reducing mean time to detect and allowing security teams to focus on critical tasks. It significantly improves visibility and threat detection across Windows, macOS, and Linux systems, enabling practitioners to detect and respond to threats in real-time. Elastic Security 7.6 also facilitates easier data ingestion from platforms like AWS CloudTrail and Google Cloud Platform, while providing enhanced visualizations and operational awareness through new features in the SIEM app, such as the Overview page and histograms. The release aims to streamline security workflows and empower security experts by making threat detection more efficient and accessible.

Kibana 7.6 introduces a range of enhancements designed to improve data visualization and analysis capabilities within the Elastic Stack. Key features include advanced formatting options for Elastic Maps, such as categorical styling and customizable text labels, and the ability to embed maps within Canvas workpads, which enhances geospatial analysis in business reporting. The release marks a significant step in machine learning integration by enabling inference using pre-trained models at data ingest, facilitating classification, regression, and outlier detection without external tools. The update also enhances Kibana Lens with support for scripted fields, allowing users to compute new fields dynamically, and introduces streamlined operations like resetting data layers. Nested field support is introduced in searches and filters, addressing a long-standing user request for more flexible data querying. Performance improvements in Elasticsearch, such as a 35x speedup in sorting dates and numbers, further enhance the efficiency of time series analysis and visualization tasks.

Organizations are increasingly migrating from Splunk to the Elastic Stack due to the latter's scalability, speed, and cost-effective open-source licensing model. Splunk's data ingest-based pricing has become cost-prohibitive as data volume and requirements for real-time insights grow, leading many companies to hit a "peak Splunk" where they can no longer efficiently scale their data operations. Elastic, originally designed for search, offers a unified stack with flexible licensing that allows users to start without purchasing a license, making it more attractive for organizations seeking to manage high-volume data sources and leverage premium features like machine learning and cross-cluster replication. The migration involves a phased approach: identifying unutilized data sources, inventorying current Splunk data, using Elastic's Beats to redirect data flows, and transferring historical data as needed. Training and consulting services are available to facilitate the transition, with examples from companies such as Box and Lyft showcasing successful switches for better return on investment and scalability.

Kibana's update for February 6, 2020, highlights recent developments and job openings related to the platform. The Maps Team has introduced a feature allowing users to style icons by category, enhancing the visualization of information on maps. Additionally, a regression issue identified in version 7.5.x, which impacted upgrades in clusters with mixed versions of Elasticsearch, has been addressed and resolved for the upcoming 7.6 release. The update also includes job opportunities for various roles such as Senior Product Designer, Senior JavaScript Engineer, and Principal Product Manager in different specializations within Kibana.

Elastic App Search, now deployable directly from Elastic Cloud, offers a robust framework for integrating efficient and scalable search functionalities into websites, applications, or mobile apps, underpinned by Elasticsearch's capabilities. It provides pre-tuned relevance settings with customizable options for refining search results, supported by comprehensive search analytics to track user interactions. The setup process is user-friendly, involving steps such as logging into Elastic Cloud, creating and customizing an App Search deployment, and indexing content through its API, which supports schemaless ingestion allowing dynamic schema generation. Users can test and improve search queries using tools like the Query Tester and Relevance Tuning, while Search UI, an open-source JavaScript library, facilitates the creation of rich search experiences with minimal coding. Additional resources, including documentation and webinars, are available to assist users in optimizing and scaling their search projects.

Elastic has announced the general availability of Elastic App Search on Elastic Cloud, offering a fully integrated, user-friendly search solution with built-in relevance tuning and analytics. Users can now easily deploy App Search instances via the Elastic Cloud dashboard, benefitting from the operational flexibility and scalability of Elastic Cloud. The new release also features a simplified resource-based pricing model, allowing users to pay only for the resources they consume, enhancing transparency and fairness. Elastic App Search is designed to be accessible to users of all skill levels, providing fast data ingestion, powerful search capabilities, and comprehensive relevance controls through a refined set of APIs and management tools. Elastic is committed to making App Search accessible for trials, prototypes, or production, offering a free 2GB instance to encourage users to get started quickly and effortlessly, making deployment, scaling, and management more straightforward for various applications.