January 2020 Summaries

In this blog post, the authors illustrate how to enhance the automation of cloud provisioning and configuration for Elastic Cloud Enterprise (ECE) using Terraform, building on a previous post that utilized Ansible for deployment. Previously, the provisioning of AWS EC2 instances and security group configuration was done manually, but Terraform now automates these processes by employing infrastructure as code, allowing for version control and programmatic management. The post provides a step-by-step guide on setting up Terraform, defining infrastructure configurations, and running the automation, specifically tailored for AWS but adaptable to other cloud providers. It highlights the simplicity and efficiency of using Terraform to manage cloud infrastructure, reducing the complexity of previous manual steps, and encourages users to experiment with provisioning and managing ECE environments. The setup is designed for a small proof-of-concept or development environment, with pointers for adapting it to full production deployments.

The announcement highlights the release of the Elastic Common Schema (ECS) .NET library, which offers a full C# representation of ECS for seamless integration with Elasticsearch, particularly for Microsoft .NET users. This library provides a structured and reliable foundation for data ingestions such as logs and metrics by defining a common set of fields. Several NuGet packages accompany this release, including Elastic.CommonSchema.Serilog, Elastic.Apm.SerilogEnricher, Elastic.Apm.NLog, and Elastic.CommonSchema.BenchmarkDotNetExporter, all of which facilitate efficient logging, distributed tracing, and performance benchmarking directly into Elasticsearch. The ECS .NET library ensures compatibility with Elasticsearch, providing out-of-the-box features for enhanced data analysis, visualization, and correlation in Kibana, while also offering a clear upgrade path via NuGet. Users are encouraged to utilize these tools to normalize their event data and leverage the full potential of ECS, with support available through the Discuss forums or GitHub.

Elastic Cloud on Kubernetes (ECK) is an operator designed to automate the deployment of the Elastic Stack, including components like Elasticsearch and Kibana, within Kubernetes environments. The process involves creating a Kubernetes cluster using Google Kubernetes Engine (GKE) across multiple zones to ensure high availability and prevent data loss. After setting up the cluster, users must configure kubectl to communicate with it and install the ECK operator. Deploying Elasticsearch clusters involves creating custom storage classes for persistent volumes and using configuration files to set up node attributes across different zones. Users can scale out the cluster by updating node configurations and can also expose the Elasticsearch service externally by configuring it as a LoadBalancer, which assigns an external IP for access. The guide concludes by encouraging users to further explore data ingestion with Elastic Cloud on Kubernetes and consider deploying additional Elastic Stack components like Kibana or Elastic APM.

The newly launched Elastic Stack community Slack workspace offers a dynamic platform for users worldwide to interact, collaborate, and stay informed about the latest developments in Elastic Stack. This initiative complements existing community engagement platforms like Discuss and Elastic{ON} Tour by providing a more immediate and interactive space for discussions. Whether users are seeking assistance with Beats modules, looking for testers for Kibana plugins, or finding local meetups, the Slack workspace serves as a hub for questions, answers, and community collaboration across various channels, including #elasticsearch, #app-search, and #logs. Additionally, the workspace features a #usergroup-organizers channel for connecting with global community organizers, encouraging more active participation in the Elastic community.

Agroknow, a Greek data and technology company, uses the Elastic Stack to enhance food safety by analyzing vast amounts of diverse, global food safety data collected through its SaaS product, FOODAKAI. This platform employs automated text mining and crawlers to gather data on food safety incidents, recalls, and pricing, which is then analyzed and made available for various stakeholders such as QA and R&D departments of food companies, and food safety researchers. The Elastic Stack, including Elasticsearch and Kibana, supports text-based queries, aggregation for visualization, and monitoring of FOODAKAI's API usage, enabling the domain experts to prioritize their review tasks based on customer needs. The platform plans to integrate machine learning to detect anomalies in data streams and improve user onboarding experiences. Mihalis Papakonstantinou, a Data Engineer at Agroknow, has been pivotal in developing FOODAKAI's data platform, which now encompasses over 65 million food safety data points.

Elasticsearch cross-cluster replication (CCR) is a feature that allows indices to be replicated across multiple Elasticsearch clusters, enhancing high availability and disaster recovery capabilities while facilitating CDN-like data architectures. Initially introduced in Elasticsearch 6.5 as a beta and later as a Generally Available feature in 6.7, CCR operates on a leader/follower model where data from a leader index is replicated to one or more follower indices, which accept no direct writes. Bi-directional replication permits seamless data access and writing across clusters without manual failover, ideal for immutable index-only workloads like logs and metrics. The tutorial provides a step-by-step guide to setting up CCR with local clusters, outlining the necessary configurations via APIs and Kibana UI, such as defining remote clusters, creating indices, and setting up read and write aliases, to ensure both data replication and high availability without requiring application changes. The CCR feature is licensed under the Platinum tier, with a 30-day trial available, and further resources are suggested for deeper exploration of CCR capabilities.

This blog post is part of a series on setting up Elastic SIEM for home and small businesses, focusing on installing and configuring Auditbeat and Packetbeat on macOS systems. The author provides detailed instructions for deploying these Beats using Homebrew, emphasizing the importance of configuring specific data collection for user activities, processes, and network data. The post highlights the challenges of integrating the GeoIP ingest processor due to issues with handling arrays of IP addresses, suggesting temporary workarounds and advising on logging configurations to troubleshoot data transmission errors. Root ownership of files is recommended to prevent unauthorized access or modification, and the setup process involves configuring network interfaces and protocols for Packetbeat, as well as ensuring the Beats are correctly installed, configured, and started. The series aims to guide users in developing a simple yet effective security solution, with future posts covering data analysis in Elastic SIEM and Elastic Maps.

Elastic has implemented its own technology through an internal analytics platform called "Vision," which leverages the Elastic Stack to provide real-time, actionable insights across various business functions. Vision utilizes the hosted Elasticsearch Service and Kibana to integrate data from multiple sources like Salesforce, Marketo, and NetSuite, offering a comprehensive view of business operations. This platform has been adopted rapidly across the enterprise, enhancing decision-making in areas from marketing and HR to sales and services. It enables teams to correlate disparate data and visualize key metrics, driving more informed data-driven decisions and solving the challenge of fragmented data systems. Through Vision, Elastic demonstrates the capabilities of its products and showcases the flexibility and power of its technology in improving internal efficiencies and strategic planning.

Elasticsearch Service has expanded its availability to the Google Cloud Platform (GCP) Mumbai region, marking its 10th GCP region globally and the third in the Asia Pacific. This expansion allows both existing and new users, who can sign up for a free 14-day trial, to access features like security, index lifecycle management, and machine learning, as well as build visualizations using Kibana Lens and Canvas. The service is designed to be user-friendly with slider bars and deployment templates, facilitating tasks such as app search, logging, metrics, APM, SIEM, and BI/analytics. Elastic partners with Google to offer the Elasticsearch Service on GCP, enhancing native GCP console integration and providing flexible subscription options through the GCP Marketplace, where usage charges are integrated into the GCP billing system. Comprehensive resources, including documentation and webinars, are available to assist users in migrating existing clusters or deployments.

This blog post, part of a series on Elastic SIEM for home and small business, provides detailed instructions on installing and configuring Auditbeat, Filebeat, and Packetbeat on a CentOS system for data collection purposes. The guide emphasizes the use of these tools to monitor user activities, processes, log files, and network data on Linux systems, and outlines the process for setting up the necessary repositories and configurations. The author also highlights the importance of configuring logging, especially when using the GeoIP ingest processor, and addresses potential issues such as the GeoIP processor's inability to handle IP address arrays, suggesting workarounds and providing links to relevant GitHub issues. Additionally, the post offers insights into configuring common Beats settings and modules, such as the auditd and system modules for Filebeat, and underscores the broader aim of the series to create a simple yet robust security solution suitable for home or small business environments.

Elastic Cloud on Kubernetes (ECK) 1.0 has moved from beta to general availability, providing users with a streamlined solution for deploying and managing the Elastic Stack on Kubernetes, a platform that has become the standard for orchestrating container workloads. ECK, built on the Kubernetes Operator pattern, addresses day 2 operational challenges such as scaling, upgrades, and configuration management, allowing users to focus on business requirements and reduce time to value. The release includes features exclusive to Elastic, like APM, Logs, Metrics, SIEM, and machine learning, all under the Basic tier, with more advanced capabilities available through an Enterprise Subscription. ECK supports a variety of Kubernetes platforms, including Google Kubernetes Engine, Red Hat OpenShift, Azure Kubernetes Service, and Amazon Kubernetes Service, and can be deployed with a simple command. The Elastic Stack also enhances Kubernetes observability and security by collecting logs, metrics, and APM data, and integrating with tools like Envoy proxy and CoreDNS. Future developments for ECK will include a dedicated UI, first-party API, and autoscaling, with updates available on their GitHub repository and Discuss forum.

In the continuation of an exploration on Mac system extensions for threat detection, the text delves into advanced techniques for obtaining detailed system event information using kernel extensions, beyond the capabilities of existing frameworks like KAuth and MACF. The discussion covers how to access process execution data such as environment variables and program arguments by navigating the inner workings of XNU, though it highlights the challenges posed by timing and access limitations. Additionally, the text examines the intricacies of intercepting Mach system calls by manipulating structures such as the mach_trap_table and mig_buckets, which involves understanding the Mach-O header format and bypassing standard kernel protections. The narrative acknowledges the impending shift away from third-party kernel extensions with Apple's transition towards the EndpointSecurity framework and System Extensions in macOS 10.16, signaling an end to the current methods and the need for adaptation to new frameworks.

An initial alpha release of a new Rust client for Elasticsearch has been announced, aimed at leveraging Rust's popularity as a fast and memory-safe programming language. This client, available on crates.io, is designed to take advantage of Rust's async-await capabilities to create efficient and reliable data interactions with Elasticsearch. The client is structured to handle the extensive range of Elasticsearch APIs by generating much of its code from Elasticsearch and X-Pack REST API specifications, simplifying maintenance and reducing errors. It employs a builder pattern for API calls, supports both JSON and NDJSON request bodies, and models API URL variants with enums to improve user experience. The project aims to align functionally with other official Elasticsearch clients and plans to evolve towards a general availability release, focusing on a user-friendly and idiomatic API that meets the Rust community's needs. Feedback is encouraged to refine the client and enhance its functionality and usability.

On January 9, 2020, Elastic announced the beta release of Elastic Cloud Control (ecctl) version 1.0.0, a command-line tool designed to interact with the Elastic Cloud Enterprise (ECE) REST API. This tool is intended for users who prefer terminal-based operations, offering an interface to create and manage deployments with features like autocomplete and an accessible help menu. The ecctl tool simplifies ECE installation management by automating tasks, and it can integrate with CI/CD pipelines, onboarding, and other scenarios. Users can install ecctl via Brew on macOS or download the binary, and configure it using the "ecctl init" command. Additionally, Elastic provides a Golang SDK, available on their GitHub, to programmatically interact with the REST API, fostering automation in Go applications. Both ecctl and the SDK are open-source under an Apache 2 license, encouraging feedback and contributions from the community.

Elasticsearch 7.5.0 introduced enhancements to its Index Lifecycle Management (ILM) that allow users to control phase transition timings by using the origination date, which specifies how old the data is rather than when it was indexed. This is particularly useful for data that arrives in Elasticsearch later than its creation, enabling more accurate lifecycle management. The feature is implemented through two settings: `index.lifecycle.parse_origination_date` and `index.lifecycle.origination_date`, which let ILM calculate index age based on the origination date. By configuring these settings, users can manage data lifecycle policies more precisely, such as making indexes read-only after a certain period and deleting them when they become irrelevant. These configurations enable better alignment of data and index ages, allowing for a unified lifecycle process for datasets that include both new and older data.

KeyBank, one of the largest banks in the United States, faced challenges with its decentralized monitoring systems, which resulted in slow issue resolution and performance-based customer loss. To address these issues, KeyBank adopted the Elastic Stack, deploying tools like Winlogbeat and Metricbeat across its network to gather log and metric data. This shift provided critical insights, revealing root causes such as high disk I/O and low memory. As the visibility into their systems improved, KeyBank undertook a system-wide refresh and continued to refine its architecture to handle growing data volumes without downtime. The iterative scaling approach, supported by Elasticsearch's horizontal architecture, enabled KeyBank to achieve rapid data ingestion and meet their service level agreements effectively. The transition not only improved system performance but also resulted in significant cost savings, illustrating the value of centralized data observability.

Andrew Cholakian's post delves into the development and optimization of a new scheduler for the Heartbeat component of Elastic's Uptime app, aimed at improving its performance in terms of reliability, speed, and resource efficiency. The initial attempt to utilize Go's goroutines and time.Timer for a minimally coded solution proved inefficient, consuming more memory and CPU. The refined approach employs a priority queue with thread-safe operations, leveraging Go's WeightedSemaphore to manage concurrent task execution and WaitGroup for sub-task synchronization. Additionally, the scheduler employs atomic operations for precise state transitions and Go's context for managing task shutdowns. This nuanced strategy ensures that the scheduler is both efficient and scalable, addressing previous challenges with high concurrency and resource management.

Kernel extensions in macOS, which allow third-party developers to enhance the functionality of the XNU kernel by dynamically loading binaries into the kernel space, provide crucial insights into security-related events such as file system, process, and network activities. These extensions, requiring Apple's special kernel extension certificate and user consent since macOS 10.14, can leverage frameworks like the Kernel Authorization (KAuth), Mandatory Access Control (MAC), and IP/Socket filter frameworks to monitor and control system events. However, with macOS 10.15 and beyond, Apple is transitioning to new frameworks such as SystemExtension, EndpointSecurity, and NetworkExtension, which will serve as the foundation for endpoint security tools. The existing frameworks offer powerful capabilities, but developers must navigate certain limitations and changes, especially with the deprecation of some features like the MAC framework in newer SDKs. The article highlights that future parts of the series will delve into the legacy frameworks' nuances and explore the newer frameworks' features and their implications for security development on macOS.