July 2020 Summaries

Elasticsearch, known for its search capabilities, can also serve as a powerful analytics engine, especially when data is structured upon ingestion using the schema on write approach. This three-part blog series explores how to enhance Elasticsearch's near real-time analytics by structuring unstructured data with the Grok Processor during ingestion. By applying grok patterns to extract specific fields from unstructured data, users can create structured documents that leverage Elasticsearch's full analytical potential. Additionally, the blog discusses the benefits of aligning data with the Elastic Common Schema (ECS) for improved visualization and automated analysis, such as machine learning-based anomaly detection. The article provides practical examples of using grok patterns in ingest pipelines, demonstrating how to parse and structure data fields like IP addresses, HTTP request methods, and more, to enable efficient analytics. It also highlights the ease of converting grok patterns between Elasticsearch's ingest node and Logstash, offering flexibility in data processing workflows.

In a presentation summarized in this post, Elastic App Search is showcased as a comprehensive and user-friendly search solution that can integrate powerful search capabilities into various applications using a set of refined APIs and management tools. The presentation, accompanied by a codelab tutorial, provides step-by-step guidance on building a movie search engine app with Python Flask, demonstrating the process of setting up an App Search instance on Elastic Cloud, data ingestion, and creating a search experience with a Python client. It highlights the ease of deploying App Search instances through the Elastic Cloud dashboard, offering the operational flexibility and scalability needed to support large deployments while leveraging the speed and relevance of Elasticsearch. The tutorial is accessible to users of almost any skill level and includes instructions for containerizing and deploying the app on a cloud platform. Additionally, users can benefit from a 14-day free trial of Elastic Cloud to explore these capabilities further, with supplementary resources available through a YouTube video and additional reading suggestions.

Elastic Cloud offers ways to optimize costs in Elasticsearch by managing replica shards, which are crucial for data resiliency and high availability. Replica shards, which are copies of primary shards, contribute to search speed but also incur additional storage and processing costs. Configuring the number of replicas can be adjusted based on the use case; for instance, reducing replicas in non-production environments or for time series data can save resources. Elasticsearch allows dynamic settings for replicas, enabling changes post-index creation, and offers automation through index templates and Index Lifecycle Management (ILM) policies to streamline the process. These tools can automatically apply settings like the number of replicas and auto-expand replicas when indices are created, providing flexibility and cost efficiency. Elastic Cloud enhances resilience by distributing replica shards across availability zones, and users can further explore these cost-saving features through a 14-day free trial.

Version 6.8.11 of the Elastic Stack has been released, featuring a series of fixes and minor enhancements to improve the stack's performance. A significant update in this patch includes Kibana's new ability to customize the SameSite cookie option, addressing bug #68108. Users are encouraged to upgrade to this latest version to benefit from these improvements. For comprehensive details on the changes made to each component of the Elastic Stack, users can refer to the full release notes provided.

Version 7.8.1 of the Elastic Stack was released on July 28, 2020, offering various fixes and small enhancements across its components. Among the notable updates, Kibana now includes support for customizing the SameSite cookie option, addressing a specific issue identified as #68108. Users are encouraged to upgrade to this latest version to benefit from these improvements. For a comprehensive breakdown of changes across products like Elasticsearch, Kibana, and Beats, the release notes provide detailed information.

"Security by default" is a critical approach highlighted in response to the "meow bot" attacks on unsecured databases, emphasizing the necessity of integrating security measures directly into technology products from the outset. These attacks exploit open databases by overwriting data with the word "meow" and numbers, showcasing the ease with which cybercriminals can target unsecured systems due to the rising demand for customer data and the availability of low-cost hacking tools. Elastic products, including Elasticsearch, combat this threat by incorporating free security features like TLS encryption and role-based access control, which are enabled by default and cannot be disabled in Elastic Cloud deployments. The post underscores the importance of vigilance in securing sensitive data, recommending the use of external scanning systems to detect exposed databases and promoting educational resources such as Elastic's free "Fundamentals of Securing Elasticsearch" course to help organizations safeguard their information effectively.

Upgrading the Elastic Stack is a vital process for administrators to benefit from performance improvements, new features, and security enhancements offered in each new version. It requires careful planning and testing, considering factors like security configurations, integration inventories, and system health to ensure a smooth transition without disruptions. The upgrade process involves understanding breaking changes and deprecation logs, enabling monitoring for resource management, and evaluating the existing system's health to avoid potential issues. Elastic provides comprehensive resources, including documentation, community support, and consulting services, to assist users in upgrading efficiently, emphasizing that while the process is manageable independently, professional support is available if needed.

Elastic Stack's 7.6 release introduced a comprehensive machine learning pipeline, integrating supervised learning capabilities alongside its existing unsupervised approaches like anomaly detection. The update enables users to create binary classification models, such as predicting telecom customer churn, by utilizing transforms to generate feature indices from raw data and employing the inference ingest processor for document enrichment. Users can develop these models without deep algorithmic knowledge, leveraging tools like data frame analytics to train models on labeled data for predictive purposes. This advancement allows for the deployment of continuous prediction systems that enhance data-driven decision-making, applicable to diverse fields such as security and observability, using Elastic Cloud's new features.

Elastic Workplace Search offers a unified search experience that enables teams to efficiently search across various content sources, such as Dropbox, Google Drive, and collaboration tools like Confluence and GitHub, from a single search bar. It simplifies the process of finding information by integrating with popular storage and productivity tools, eliminating the need to remember where specific documents are stored. The tool is designed for easy implementation, allowing organizations to get it up and running within days and includes features like relevance tuning and content source prioritization through simple sliders, without requiring IT involvement. This search solution enhances productivity by allowing teams to tailor search results to their specific needs, ensuring that the most relevant content is prioritized according to team preferences, such as design or engineering. Elastic Workplace Search also supports the integration of other data sources using the Custom Source API, providing the same automatic filtering and customizable results display.

Elastic Cloud offers enhanced cost management and optimization strategies through the use of hot-warm architecture and index lifecycle management (ILM) in Elasticsearch Service. This approach allows users to manage data efficiently by categorizing it into different storage tiers, with high-performance SSDs for frequently accessed data in the hot phase, and more cost-effective storage for less frequently accessed data in the warm phase. The ILM automates data management actions such as rollover and force merge, leading to significant cost reductions, potentially up to 60%. Users can create ILM policies in Kibana and assign them to indices via templates, facilitating automated transitions between data phases. This strategy not only optimizes storage costs but also maintains the capabilities of the Elastic Stack for data visualization, alerting, and anomaly detection. Elastic Cloud encourages users to adopt these practices to realize substantial cost savings and improve data management efficiency.

Luis Francisco Sánchez Merchante discusses the transformative impact of the COVID-19 pandemic on education, emphasizing the shift towards remote learning and the integration of cloud technologies. He highlights the necessity for educational institutions to adapt by collaborating with IT companies to incorporate up-to-date digital tools and infrastructure, which are crucial for maintaining education quality and relevance in a rapidly changing world. Merchante illustrates this with the partnership between Elastic and Comillas University, where students are trained in data visualization using Elastic Cloud, enabling them to gain practical experience with industry-standard tools. This collaboration not only enhances the students' learning experience but also prepares them for data-centric roles across various fields, illustrating the increasing importance of integrating technology and education to produce well-equipped graduates.

In the blog post, Dale McDiarmid outlines the process of integrating Trello data into Elastic Workplace Search using the Custom Source API. This integration enhances the accessibility and searchability of Trello's Kanban-style boards, which are valuable for tracking tasks and facilitating organizational collaboration. The guide provides a step-by-step explanation of setting up a Python script to ingest Trello data, such as team boards, cards, and comments, into Workplace Search, emphasizing the importance of metadata and the need for synchronization and security considerations. It illustrates how to configure Trello and Workplace Search APIs for data ingestion, create a custom content source, and adjust schemas and display settings for optimal search results. The post notes that while the example focuses on a limited dataset, it opens possibilities for cross-team collaboration and reducing duplicated work. Additionally, it highlights potential security complexities regarding access permissions, which will be explored in future discussions.

The Python Elasticsearch client has introduced native async I/O support in its version 7.8.0, addressing growing demand due to the rise of asynchronous Python web frameworks like FastAPI, Starlette, and the upcoming Django 3.1. Async I/O offers notable performance improvements for I/O-heavy workloads, such as web applications, by using system resources more efficiently than traditional multi-threaded applications. This update also includes support for the new Elasticsearch 7.8 APIs. To utilize this feature, users need to install the package with the [async] extra, which incorporates Aiohttp for making HTTP requests to Elasticsearch instances. The updated client allows developers to integrate async operations seamlessly, demonstrated through examples using IPython for immediate event loop access. Additionally, comprehensive documentation, including examples and a webinar scheduled for August 5th, provides further insights into the client's capabilities and integration with Elastic APM.

In this comprehensive tutorial on Kubernetes observability, the focus is on monitoring application performance with Elastic APM, which is part of the Elastic Observability suite. The guide emphasizes using Elastic APM for Application Performance Monitoring (APM), enabling the automatic measurement of key service-level indicators like request/response latency and errors, thereby facilitating rapid identification of performance issues. It highlights features such as distributed tracing, which measures the end-to-end latency of distributed components, and the integration of APM with logs and metrics to provide a holistic view of application health within Kubernetes. The tutorial also covers the deployment of APM agents alongside application components, the use of Java and Real User Monitoring (RUM) agents, and the visualization of custom metrics with Kibana’s Lens tool. Additionally, the significance of log correlation with APM traces is discussed, showcasing how the Elastic APM Java Agent enriches log data with trace identifiers, allowing for seamless integration and analysis of observability data. The series concludes by suggesting additional Elastic components like Heartbeat and Packetbeat to further enhance observability and offers resources for setting up and troubleshooting the monitoring system using Elastic Stack.

Elastic Cloud has introduced new traffic management features, including IP filtering and integration with AWS PrivateLink, to enhance network security for its deployments on AWS, Google Cloud, and Microsoft Azure. These features allow users to control access to their Elastic Cloud deployments by specifying network access based on IP addresses or ranges and by enabling private connectivity between virtual private clouds (VPCs) and on-premises applications through AWS PrivateLink. This integration supports secure communication using private IPs, simplifies network management, and aligns with security practices such as the principle of least privilege and layered security. Users can create and manage traffic filters in the Elastic Cloud console, associating them with deployments to restrict access to trusted sources, thereby securing sensitive data and personally identifiable information. These enhancements are available to all Elastic Cloud customers at no additional cost, and detailed guidance for setup is provided in the product documentation.

Elastic has announced the availability of Elastic Workplace Search on Elastic Cloud, offering unprecedented deployment flexibility across major cloud platforms like AWS, Google Cloud, and soon Azure. This allows users to choose their preferred cloud infrastructure, avoiding vendor lock-in and enabling them to leverage existing cloud agreements. Elastic Workplace Search, built on Elasticsearch, provides high performance and speed with low latency due to its global availability in 37 regions, ensuring efficient internal search capabilities. The deployment is simplified through Elastic Cloud’s management tools, offering easy scaling and resource-based pricing, which charges only for consumed resources, freeing users from traditional licensing models. Users can quickly initiate a free trial and expand their deployment without significant upfront investments.

The blog post explores the differences between macOS and Windows in handling system-level events crucial for endpoint security analysis, particularly focusing on file and network events. For file events, Windows utilizes a powerful and flexible minifilter driver that offers granular control, while macOS employs the Endpoint Security (ES) framework, which simplifies usage but limits developer control, requiring specific entitlements from Apple. In network event handling, Windows provides the Windows Filtering Platform (WFP) for comprehensive network traffic filtering across various layers, whereas macOS has transitioned from Network Kernel Extensions (NKE) to the more simplified and restricted Network Extension framework. The discussion highlights Windows' approach of empowering third-party developers with extensive tools and Apple's strategy of maintaining control by offering highly wrapped APIs, emphasizing the importance of understanding these differences for effective cybersecurity practices.

DevSecOps is an integrated approach that combines development, security, and operations to enhance software development and deployment speed, reliability, and security. As the distinction between these roles becomes increasingly blurred, the shift-left mentality encourages collaboration from the outset, integrating security into the agile development process to create more secure systems. The Elastic Stack plays a crucial role in supporting DevSecOps by unifying and analyzing data across the entire stack, offering features such as fast search capabilities, automated data distribution, policy-driven data lifecycle management, and observability. This integration allows for comprehensive monitoring and analysis of CI/CD pipelines, application performance, and security, while also facilitating seamless deployment and container orchestration. Elastic's tools and solutions enable organizations to achieve continuous integration, continuous delivery, and continuous monitoring, fostering an environment of agility and robustness in software development practices.

In the second part of this blog series, the use of the Elastic Stack's machine learning capabilities is explored to detect Domain Generation Algorithm (DGA) activity in network data. By employing a supervised classification model, network data can be enriched with classifications during ingestion, identifying potentially malicious domains by analyzing DNS queries. The methodology involves setting up an ingest pipeline with inference and Painless script processors to extract features like unigrams, bigrams, and trigrams, which are then used by a pre-trained model to predict domain maliciousness. Addressing false positives, which can be substantial due to high DNS traffic volumes, the blog suggests using anomaly detection as a secondary analysis technique to differentiate actual DGA activity from noise. This approach not only improves the accuracy of threat detection but also demonstrates how the Elastic Stack can be configured to enhance cybersecurity measures through automated data enrichment and analysis. For practical application, the blog provides guidance on setting up and testing these systems using Elastic's services, offering a trial for users to experiment with these tools in their network environments.

In the second part of a Kubernetes observability tutorial series, the article focuses on using Elastic Observability to monitor applications running in Kubernetes by collecting and analyzing various metrics. It highlights the challenges involved in monitoring Kubernetes due to its distributed nature and the variety of technologies involved. The tutorial introduces Metricbeat, a tool used to collect metrics from Kubernetes pods and clusters, and explains how it integrates with Elastic's Kibana for data visualization. Metricbeat can be deployed in two ways: as a single pod for cluster metrics using kube-state-metrics and as a DaemonSet for host-specific metrics via the kubelet API. The tutorial also covers using Metricbeat's autodiscovery feature to apply technology-specific modules for collecting metrics, with examples using NGINX and MySQL. Furthermore, it describes how Metricbeat can augment or replace Prometheus for application metrics collection, facilitating the correlation of metrics with logs and application performance monitoring data. The article concludes by encouraging readers to explore Kibana's pre-built dashboards and to utilize Elastic's services for a comprehensive observability solution.

Elasticsearch is widely used to power search experiences across various applications, and optimizing the connection between the application and the Elasticsearch cluster is crucial for user experience. Sniffing is a technique that can enhance this connection by allowing clients to update their connection pools dynamically, based on the current state of the cluster. It involves accessing the _nodes/_all/http endpoint to retrieve a list of nodes, but it is not without challenges, particularly when dealing with different network configurations or cloud providers. Sniffing is beneficial when Elasticsearch clusters are within the same network as the client, but issues may arise if the cluster is behind a load balancer or resides in a separate network. In such cases, configuring Elasticsearch to bind to its host while advertising another address can be a solution, or using a proxy to manage node failures. Sniffing strategies include executing it at startup, on connection failure, periodically, or through custom configurations, each with its advantages depending on the infrastructure. However, it is important to understand the specific setup before implementing sniffing, as it may not always be the optimal approach, especially when using services like Elastic Cloud, which handles these complexities internally.

In response to the challenges posed by COVID-19, the Elastic DOD team organized a virtual hackathon aimed at fostering collaboration within the U.S. Air Force's Mission Defense Teams (MDT). Held from June 10 to June 18, the event brought together 20 teams tasked with creating innovative projects using pre-built datasets and clusters, with a focus on data ingestion, dashboard creation, and alerting. The top three projects showcased the participants' ingenuity and technical prowess. The 601 AOC MDT team won first place by developing tools to extract complex firewall datasets for better network security, while the 673 CS MDT team utilized Raspberry Pis to evaluate Wi-Fi signals and enhance security measures. The 553 ACNS MDT team streamlined data access by integrating multiple datasets into a single, user-friendly interface. The success of this hackathon has prompted plans to extend similar initiatives to other DOD teams, with ongoing opportunities for training and participation.

The text examines the differences between Windows and macOS operating systems, particularly in how they handle security events, focusing on process events in Part 1 of a two-part series. It highlights that despite both systems evolving into hybrid kernels, their foundational differences, such as Windows' open ecosystem versus macOS's closed, integrated approach, result in distinct development environments. Windows offers abundant APIs and documentation, facilitating flexible development, while macOS's approach emphasizes simplicity and control, which can limit third-party access. The discussion also touches on kernel architecture, with macOS moving away from kernel extensions to user-mode system extensions, contrasting with Windows' encouragement of user-mode driver implementation. Furthermore, the text explores how process events are managed, noting Windows' use of kernel callbacks for process notifications, while macOS relies on the Endpoint Security framework to handle process events with improved information offerings like command-line arguments and code-signing data. The shift in macOS from kernel extensions to system extensions illustrates its move towards a more controlled, microkernel-like environment, affecting how developers interact with the system.

On July 1, 2020, Elastic Cloud announced the support for Amazon EC2 M5d and R5d instances across all AWS regions, enhancing performance and flexibility for various workloads. The M5d instances offer a balanced combination of compute, memory, and networking resources ideal for tasks like Elastic App Search and Elasticsearch machine learning nodes, while R5d instances are optimized for handling large data sets in memory, suitable for application performance monitoring and Kibana. Both instance types support locally attached NVMe SSD drives, providing superior disk performance compared to EBS General Purpose GP2 SSD volumes, with the option to either launch new deployments or migrate existing ones through a straightforward snapshot and restore process. Elastic Cloud provides deployment templates to streamline hardware selection, ensuring configurations are tailored to workload requirements, and encourages users to explore these new capabilities through their migration documentation and a free 14-day trial.

Analyzing online search relevance metrics involves understanding how well a search experience satisfies a user's information needs by examining various user interactions and behaviors. This process starts with capturing events like queries, page views, and clicks from users interacting with a search application and ingesting these events into Elasticsearch. Once collected, these events are transformed into per-query relevance metrics, which can be aggregated over time to provide insights such as click-through rates and query distribution. The Elastic Stack tools, including Kibana and Elasticsearch, assist in visualizing and managing these metrics, allowing for fine-tuning and improving search relevance. While metrics are helpful, they carry biases and require additional sophisticated tools for deeper analysis, like A/B testing. The post encourages further exploration and customization of metrics while discussing the potential challenges and limitations of using real user behavior as a basis for measuring search relevance.

In 2019, the Kibana platform team initiated a significant re-architecture project to address the growing complexities and limitations of Kibana's existing structure, which had expanded from three applications to over 100 plugins since 2015. This migration aimed to enhance Kibana's stability, scalability, and performance by restructuring millions of lines of code and collaborating with more than 15 engineering teams to transition all first-party plugins to the new platform, a task that was successfully completed by June 2020. The project emphasized building a collaborative culture through clear communication of project goals, establishing a detailed project charter, and defining clear roles and responsibilities to ensure coherence and accountability across teams. It also highlighted the importance of standardized processes, effective knowledge sharing, and regular progress tracking to manage the complexities of cross-team collaborations. Recognizing the need for motivation throughout the lengthy process, the team celebrated key milestones to maintain momentum and acknowledged the extraordinary collaboration and professionalism of all contributors.

Elastic Cloud has introduced several new features aimed at enhancing user experience and flexibility, including self-service subscriptions and in-place configuration changes. Users can now purchase Gold and Platinum monthly subscriptions directly through the Elastic Cloud console, allowing access to exclusive capabilities such as enterprise search, observability, and security, with a resource-based pricing approach detailed on their pricing page. The platform also provides the option to change subscription levels as business needs evolve, offering flexibility through monthly billing and deeper discounts for annual commitments. In-place configuration changes facilitate faster and more reliable updates by applying changes directly to the cluster, avoiding lengthy data migrations. Additionally, Elastic Cloud now supports AWS M5d and R5d instance types, which offer improved disk performance, and users can sign up using their Google Account for easier access. New users are encouraged to explore Elastic Cloud with a free 14-day trial.