August 2020 Summaries

The official Go client for Elasticsearch, developed by Elastic, offers a structured approach to interacting with Elasticsearch's APIs and sending data to and from the cluster, having evolved since its initial release in early 2019. The client architecture effectively separates concerns by dividing functionality into distinct packages: esapi for API interactions and estransport for data transport, enabling maintainability, extensibility, and flexibility. The design philosophy emphasizes providing a robust out-of-the-box experience while allowing users to adapt the client for specific needs, demonstrated by features like custom connection pooling and node discovery. The esapi package offers a method-oriented API, facilitating easier code readability and parameter management, while the estransport package handles network operations, connection management, and request retries. The client also supports advanced configurations such as custom connection selectors and node discovery, enhancing adaptability in diverse environments. The client maintains consistency across languages by adhering to Elasticsearch's API specifications, with automatic code generation ensuring alignment with the evolving API. Performance considerations lead to leaving encoding and decoding tasks to the calling code, allowing integration with third-party JSON packages for improved efficiency. Future blog posts will delve into client configuration and customization, building on this foundational overview.

Elastic has introduced new Quick Start guides for its Enterprise Search products, Elastic Workplace Search and Elastic App Search, built on the Elastic Stack to provide fast, scalable, and relevant search solutions without extensive development efforts. Each guide is a 15-minute video offering a comprehensive overview and step-by-step instructions to help users create effective search experiences for workplaces, websites, and apps. Elastic Workplace Search integrates with various productivity tools, while Elastic App Search is designed for diverse applications such as ecommerce and job portals, with features like refined APIs and robust analytics. Both guides demonstrate setting up deployments using a free Elastic Cloud trial, synchronizing data, and enhancing search functionalities. Additional resources, including webinars and documentation, are available for users seeking more in-depth guidance, and free Elastic Cloud trials are accessible to all users without a credit card requirement.

This blog post provides a detailed guide on configuring a Security Assertion Markup Language (SAML) realm for role-based access control (RBAC) in Elastic Cloud Enterprise (ECE) environments using Auth0 as an identity provider. It begins by outlining the prerequisites, including the necessary ECE and Auth0 versions, and follows with step-by-step instructions for setting up the SAML realm. The process involves creating a new application in Auth0, configuring SAML settings to align with ECE requirements, and mapping user attributes and roles to ensure seamless integration. The post also highlights the importance of troubleshooting and adjusting logging levels to manage SAML authentication effectively. Finally, it emphasizes the benefits of using SAML for ECE RBAC, allowing users to log into the platform using an external identity provider, thus simplifying administration and enhancing security.

Alex Marquardt's article provides a step-by-step guide on incrementally building custom grok patterns for structuring unstructured data in Elasticsearch, specifically targeting faster analytics with schema-on-write. Utilizing tools like the Kibana Grok Debugger and the simulate pipeline API, the article demonstrates how to construct a grok pattern from scratch, using an example message to extract ECS-compliant fields such as IP address, HTTP request method, URI, request size, and event duration. By starting with simple patterns and progressively adding complexity, the guide illustrates how to address common pitfalls, such as missing spaces, ensuring that each component of the message is correctly parsed. The article concludes by encouraging readers to experiment with creating their own custom ingest pipelines and teases future content on debugging grok expressions.

Elastic Cloud has achieved FedRAMP authorization at the Moderate Impact level on AWS GovCloud (US), enhancing its appeal to federal organizations and customers in regulated sectors by providing speed, simplicity, and security. New and existing users managing data for government entities or educational institutions can sign up for Elastic Cloud on AWS GovCloud, allowing them to deploy Elastic Cloud clusters, access Kibana, and secure connections to advance search, observability, and security projects. Users can create a deployment, access data through Kibana, and enhance network security using AWS PrivateLink and IP filtering. Elastic Cloud offers powerful tools for enterprise search, observability, and security, including Elastic App Search, Workplace Search, and pre-built detection rules in the SIEM app, facilitating efficient data management and threat detection. The platform supports scalable data management with hot-warm templates and lifecycle management features, and users are encouraged to explore Elastic's documentation and resources for further learning and subscription options.

This blog post, aimed at security practitioners, provides an in-depth understanding of Windows access tokens, crucial for detection engineering. It explains the relationship between logon sessions and access tokens, emphasizing that access tokens act as proxies for logon sessions, storing security settings and privileges. The article details how access tokens are linked to logon sessions via a 64-bit locally unique identifier and how these tokens can be adjusted without affecting other processes. It also covers network authentication, highlighting the role of access tokens in impersonation, where different threads can assume different security contexts. The post notes that impersonation is used to handle client requests in multi-threaded applications, emphasizing the importance of understanding these mechanisms to detect and prevent token manipulation. The piece concludes with a preview of the next part in the series, which will explore how attackers exploit Windows functionality to move laterally and compromise Active Directory domains.

The 7.9 release of Elastic Enterprise Search introduces significant enhancements aimed at providing powerful and modern search experiences for app users, workplace teams, and website visitors, with notable updates to both App Search and Workplace Search. Key features include the addition of Workplace Search features to the free distribution tier, offering connectors to popular productivity tools and a Custom Source API, which allows users to boost productivity with modern search experiences without incurring additional costs. The release also integrates Elastic Enterprise Search into Kibana for seamless navigation and management, introduces a federated Gmail connector for Workplace Search, and includes advanced document tracking and logging for enhanced organizational insights. Moreover, the update inherits index lifecycle management policies from the Elastic Stack for automatic log and analytics management, while new features like document thumbnail previews and personalized search results based on user behavior enhance the overall user experience. Existing and new users can access these features through Elastic Cloud or by downloading the self-managed versions.

Elastic Cloud's 7.9 release introduces several key updates, including enhanced network security with AWS PrivateLink and IP filtering support, compliance with FedRAMP Moderate impact level for secure use by federal agencies, and the availability of Elastic Cloud in new regions. The update also offers streamlined purchasing options through self-service monthly subscriptions and improved service performance with in-place configuration changes and new AWS instance types. Additionally, Elastic Cloud Enterprise 2.6 and Elastic Cloud on Kubernetes 1.2 are now generally available, providing enhanced deployment capabilities for Elasticsearch clusters and enterprise search solutions, alongside a unified command-line interface for managing deployments. These advancements aim to bolster security, flexibility, and performance for users across various cloud environments.

Logstash 7.9.0 has been officially released, bringing significant optimizations and new features that enhance performance and expand use cases. The update includes a dramatic reduction in pipeline startup and restart times, with over 9x faster compilation, greatly benefiting pipeline development workflows. The Elastic App Search output plugin has been revamped with a robust Ruby client library, expanding its integration capabilities and offering a smoother migration path from custom Elasticsearch applications. The release also extends API key support across various Logstash and Elasticsearch plugins, improving security and management capabilities. Additionally, Logstash now supports ARM/AARCH64 systems, although this support remains experimental and not recommended for production. Users are encouraged to try the new version, report feedback, and explore the full range of supported operating systems.

Kibana 7.9 introduces a significant architectural overhaul aimed at enhancing performance and productivity, offering faster page loads and streamlined development capabilities for users and developers. The release features new dashboarding capabilities, enhancements to Kibana Lens for data visualization, and expanded mapping and machine learning functionalities. Notably, it integrates Elastic Enterprise Search into Kibana, offering quick access to App Search and Workplace Search engines. New features like the "explore underlying data" functionality allow users to delve deeper into data directly from visualizations, while improvements in Anomaly Explorer visuals increase dashboard interactivity. Additionally, the update includes enhanced iframe configurations, support for multiple y-axes comparisons in Kibana Lens, and custom color selectors. Kibana 7.9 also addresses data gap handling with fitting functions, facilitates efficient Canvas workpad updates through variables, and expands Elastic Maps with new prebuilt map layers and custom vector data sources. The alerting framework now integrates with ServiceNow, and machine learning capabilities are augmented with features like model snapshot management and automated anomaly detection annotations. These advancements aim to provide deeper insights, faster data analysis, and improved user experiences across the Elastic Stack.

Elastic Observability 7.9 introduces several significant enhancements aimed at simplifying data onboarding and improving security and monitoring capabilities. The release features a unified Elastic Agent that consolidates data collection across logs, metrics, and endpoint security, reducing the complexity of managing multiple agents. Users can manage these agents centrally with Fleet, and the new Ingest Manager UI simplifies configuration, moving away from traditional YAML files. Additionally, the new observability overview page in Kibana provides a comprehensive view of system health by unifying logs, metrics, and traces. Security is bolstered with 50+ prebuilt detections for AWS and Okta, enabling rapid threat signal detection. The release also integrates OpenTelemetry, allowing seamless migration of observability data, and enhances Azure monitoring. These updates aim to reduce operational overhead, improve security, and accelerate insights for observability teams.

Elasticsearch 7.9.0, based on Lucene 8.6.0, introduces several enhancements and new features, including improvements to data ingestion, search capabilities, and machine learning functionalities. The release simplifies the ingest process with data streams, allowing efficient handling of time series data across multiple indices, and introduces a flexible memory-based queue to improve ingestion throughput. It also enhances search capabilities with the new wildcard data type and introduces Event Query Language (EQL) for advanced threat detection. The update includes new aggregation functions like variable width histogram and moving percentiles aggregation, along with performance optimizations. Additionally, the release features a Tableau Connector, enabling direct access to Elasticsearch data from Tableau, eliminating the need for ETL processes. These updates are aimed at making Elasticsearch more efficient and versatile for users across various domains, including security and observability.

Elastic Security 7.9 introduces significant advancements in endpoint security by integrating signatureless malware prevention and kernel-level data collection into the Elastic Agent, enhancing data collection and threat prevention capabilities. The release includes prebuilt cloud protections, new detection rules aligned with the MITRE ATT&CK framework, and machine learning jobs to improve cloud security posture monitoring, addressing misconfiguration risks identified in Verizon’s 2020 DBIR. The update also features community-driven workflow refinements, including threshold-based alerting, building block rule designation, and customizable investigation timeline templates to streamline security operations. Additionally, Elastic Security 7.9 supports new data integrations with Microsoft Defender ATP, Windows PowerShell, and Sophos XG firewalls, among others, and includes enhancements to the user interface to improve analyst efficiency and reduce false positives. The platform ensures consistent task execution with updates to support Elastic Common Schema 1.5, offering expanded data collection and preparation capabilities to support a variety of use cases for security teams.

Elastic 7.9 introduces a broad array of enhancements across its Enterprise Search, Observability, and Security solutions, all built on the Elastic Stack. Key features include the introduction of a free distribution tier for Workplace Search and endpoint security, simplifying data onboarding with the Elastic Agent and Ingest Manager, and enhanced observability through a unified overview page in Kibana. The release strengthens integration with open standards like OpenTelemetry and expands capabilities in Elastic Security with new detection rules and integrated malware prevention. Users can now benefit from faster navigation in Kibana due to architectural improvements and explore the new Event Query Language (EQL) for threat detection. Elastic Cloud has been improved with support for AWS PrivateLink, offering enhanced security and compliance options, and expanded regional availability. Additionally, Elastic Cloud Enterprise 2.6 and Elastic Cloud on Kubernetes 1.2 bring new management capabilities and ease of deployment for Elastic solutions.

Version 6.8.12 of the Elastic Stack has been released, featuring fixes and minor enhancements across the stack. Users are encouraged to upgrade to this latest version to benefit from these improvements. Detailed information on specific changes for each product within the stack can be found in the release notes.

Elasticsearch 7.9 introduces a new "wildcard" field type designed to enhance the efficiency of finding patterns within string values, particularly beneficial for indexing and searching logs and security data. This field type simplifies search expressions and indexing processes, offering faster searches and reduced disk usage without altering query syntax. Unlike traditional text and keyword fields, the wildcard field excels in partial matches and infix searches while eliminating size limitations and improving performance for high-cardinality fields. It employs an "n-gram" indexing method and binary doc value store to quickly narrow down search candidates, making it a suitable replacement for keyword fields in certain scenarios. The new field type is expected to be integrated into the Elastic Common Schema, potentially offering faster search capabilities in future updates without requiring changes to client applications.

Elastic Cloud is now available on Amazon Web Services (AWS) in the Asia Pacific (Seoul) region, providing enhanced capabilities in enterprise search, observability, and security. Users can search various platforms, monitor applications, analyze logs and metrics, and implement endpoint security. Elastic Cloud offers flexibility in deployment, allowing users to choose between managed services or self-management on AWS, Google Cloud, or Microsoft Azure, and to optimize deployments based on specific requirements. Integration with the AWS Marketplace allows AWS customers to subscribe to Elastic Cloud and Elasticsearch Service, with usage charges consolidated into their existing AWS bills. Customers can easily start by logging into the Elastic Cloud console or signing up for a free trial.

Elastic Cloud has expanded its availability to the EU (Paris) region on Amazon Web Services (AWS), providing users with enhanced enterprise search, observability, and security capabilities. This expansion allows users to easily search applications and websites, monitor application performance, analyze centralized logs, and enhance endpoint security. Elastic Cloud offers flexible deployment options, allowing users to choose between a managed service or self-management, and supports orchestration on AWS, Google Cloud, or Microsoft Azure. Users can optimize deployments based on specific compute, memory, or I/O requirements and quickly provision workloads through the Elastic Cloud console. Additionally, AWS customers can subscribe to Elastic Cloud services through the AWS Marketplace, with integrated billing options that facilitate seamless financial management and include deductibles from prepaid AWS Enterprise Discount Program agreements.

In August 2020, members of the Elastic Security team hosted a virtual threat hunting capture the flag (CTF) event at BSides SATX, providing an opportunity for participants to practice threat hunting using Elastic's free and open technologies. The event involved using version 7.8 of the Elastic Stack to sift through 11 million events in search of malicious behavior, simulating a common scenario where a business partner's network had been compromised by ransomware. Participants were tasked with forming hypotheses and searching for suspicious activity without preset security alerts, using tools like Kibana for visualization and query execution. The CTF environment was configured with Windows Server and Windows 10 endpoints, employing applications like Winlogbeat and Packetbeat to generate network and log data. By engaging in this exercise, participants learned to navigate Elastic's data schemas and enhance their threat detection abilities. The event highlighted Elastic's broader philosophy of security, emphasizing the importance of reliable data and technology-aided processes for effective threat hunting and security operations.

In the realm of containerized environments and Kubernetes, securing sensitive information while maintaining observability is paramount. Metricbeat, with its autodiscover feature, provides a flexible solution for monitoring dynamic Kubernetes workloads by adapting configurations as services change. It offers two strategies: template-based and hints-based autodiscover, allowing users to set up monitoring with minimal permissions. However, securing credentials remains a challenge, especially when services require authentication. To address this, Metricbeat introduces the keystore feature, enabling secure storage of passwords, thus avoiding plain text or environment variables. While template-based autodiscover can utilize the Metricbeat keystore, hints-based autodiscover requires the Kubernetes Secrets Keystore, which allows consumption of Kubernetes secrets directly in configurations, ensuring sensitive information is kept secure within the same namespace. This approach enhances security by preventing password exposure and enables secure connections for autodiscovered workloads, with Elasticsearch Service on Elastic Cloud providing an additional layer of security.

Michael Young's blog post provides a comprehensive guide on configuring Filebeat and Zeek to analyze network data using Elastic Security. It details the process of enabling the Zeek module in Filebeat, configuring the Zeek logs to output in JSON format, and using Filebeat modules to parse and visualize data. The blog emphasizes the importance of customizing configurations to suit specific environments and highlights the steps required to correctly display Zeek data on Elastic Security maps through GeoIP enrichment. Additionally, it addresses common issues related to the data flow timing between Filebeat processors and the ingest pipeline, offering solutions to ensure accurate data representation. The guide encourages readers to explore further data integrations with Elastic Security, such as incorporating Suricata and host data streams, to enhance their security analytics capabilities.

The UK's Driver and Vehicle Licensing Agency (DVLA) is modernizing its IT infrastructure by embracing cloud-based platforms and employing the Elastic Stack to centralize logs, thereby enhancing observability and performance monitoring across its extensive operations. This initiative, part of the DVLA's Open Services Landscape strategy, aims to address interoperability challenges by allowing data to be shared seamlessly across different systems, ultimately improving customer experience and operational efficiency. To support this transformation, the DVLA has invested in Elastic Training, equipping its teams with crucial skills in utilizing the Elastic Stack, fostering a collaborative learning environment, and setting the agency on a path to become a center of excellence for cloud services within the government. The training has enabled teams to create and manage dashboards independently, and the DVLA is exploring further applications of Elastic’s offerings, such as Elastic Security, to enhance their IT capabilities.

The blog post by Tom Lawler and Nicole Albee discusses strategies for optimizing costs in Elastic Cloud, focusing on availability zones and snapshot management. It emphasizes the importance of configuring deployment size and settings to suit specific use cases, such as adjusting snapshot frequency to manage storage costs and understanding how to pause deployments effectively. The text highlights the balance between cost and resiliency, suggesting the reduction of availability zones to cut costs while acknowledging the increased risk of data loss. For snapshot management, it details the benefits of using Snapshot Lifecycle Management (SLM) for Elasticsearch versions 7.6 and later, while older versions require the Elastic Cloud UI. The blog also describes how to configure snapshots to suit different needs by adjusting frequency, retention, and storage location, which can significantly impact costs. The authors encourage readers to explore previous installments of their series for further tips on cost management in Elastic Cloud and promote Elastic Training courses for more advanced management techniques.

The article by Jeff Vestal introduces Elastic Uptime, a tool powered by Heartbeat, designed to simplify the monitoring of IT service availability. It explains the significance of availability in various IT contexts, such as websites, networks, API endpoints, and development productivity, and describes how Elastic Uptime can track and provide insights into these areas using Kibana. The interface of Elastic Uptime is detailed, highlighting its ability to provide an overview of service statuses, monitor certificates, and prevent costly outages due to expired certificates. Heartbeat, as a lightweight data shipper, is essential for actively probing services to ensure operational availability. The article also provides a practical example of monitoring HTTP availability using Heartbeat across various geographical locations, demonstrating how to configure and deploy it on Elastic Cloud. It emphasizes the integration of uptime data with other observability data like logs and metrics, which helps create comprehensive visualizations and facilitates quicker detection of issues. The piece concludes by hinting at future topics, including anomaly detection and alerting, encouraging readers to explore Elastic Uptime further.