November 2019 Summaries

Elastic has undertaken a comprehensive effort to ensure pay equity by working with Economists Incorporated to analyze compensation practices. The initial study revealed that women at Elastic, both in the US and globally, earned nearly the same as their male counterparts in similar roles, with women receiving slightly higher equity awards, though not statistically significant. Following a mid-year compensation review, Elastic identified a few pay discrepancies, most of which were explainable, and corrected those that weren't. The company also incorporated age into its analysis, finding equitable practices and plans to add ethnicity data in the next review cycle. Elastic is committed to building an equitable workplace by continuously examining pay practices across gender, age, and soon ethnicity, particularly considering the impact of race on pay equity in the US, where a significant portion of its workforce is based.

Kevin Kluge shares his journey of joining Elastic, highlighting its unique distributed work culture and engineering ethos. Initially intrigued by the potential of Elastic's product, Kluge was drawn to the company's emphasis on work-life balance, allowing him to prioritize family time without sacrificing career growth. Elastic's early days featured a small, geographically dispersed team of engineers, fostering a culture of self-motivation and autonomy. This approach has been maintained as the company scaled, emphasizing effective communication across time zones and creating an inclusive environment. Elastic's organizational structure and communication practices promote equality and collaboration, with a focus on results rather than micromanagement. As the company grows, it continues to offer career development opportunities for engineers, ensuring that values like empowerment and flexibility remain central to its culture.

Elastic Security combines Elastic SIEM with Elastic Endpoint Security to create a solution aimed at detecting and responding to security threats, leveraging machine learning for real-time protection and threat detection. Machine learning is crucial in information security for its automation capabilities, discovery of sophisticated relationships, and ability to generalize less brittle detection patterns compared to hand-crafted rules. Elastic's MalwareScore utilizes machine learning to identify malicious files by analyzing features from vast datasets of malicious and benign files. Challenges in using machine learning for security include false positives, explainability, and adversarial drift, which Elastic addresses through continuous model retraining and engineering. The MITRE ATT&CK framework is used to map adversary tactics, and Elastic's approach integrates machine learning to enhance detection and automate security processes. Future developments will involve using machine learning in Elastic SIEM for anomaly detection, providing broader security insights and threat detection.

In a lawsuit filed on September 4th, Elastic, the company behind Elasticsearch and Kibana, accused floragunn GmbH of copying proprietary code for its security plugin, Search Guard, and has since expanded the claim to include additional infringements pertaining to Kibana. The updated lawsuit identifies further copying by floragunn in the Search Guard Kibana plugin, notably in user management features, and highlights third-party products and services that utilize the infringing code, such as Amazon Elasticsearch Service and IBM Cloud Databases for Elasticsearch. Although these companies are not being sued, their use of Search Guard places them in a complex legal situation. Elastic emphasizes that its products offer free security features and encourages Search Guard users to consider these options to avoid running unprotected clusters. Elastic invites users with concerns to reach out for support and clarification.

Elasticsearch Service has expanded its availability to the Azure Washington (westus2) region, marking its fourth global Azure region and the second in the United States. This expansion allows existing users to access the new region immediately via Elastic Cloud, while new users can explore it with a 14-day free trial. The service includes exclusive features such as native security, index and snapshot lifecycle management, and machine learning capabilities, along with support for various use cases like logging and analytics. Elastic, in partnership with Microsoft, aims to enhance the Elastic Stack experience on Azure and plans to introduce more Azure regions for the service. Additionally, customers will soon be able to purchase Elasticsearch Service subscriptions through the Azure Marketplace, with comprehensive migration support available for those transitioning existing deployments.

Elasticsearch Service is now available on Microsoft Azure in the Singapore region, marking the first Azure region in Asia for Elastic Cloud, which now has four global Azure regions. Existing users can access the service immediately, while new users have the option to sign up for a 14-day trial. The service offers unique features such as native security, index lifecycle management, and machine learning, and supports various use cases including logging, metrics, APM, and BI/analytics. Elastic continues to expand its partnership with Microsoft, aiming to provide Azure customers with the best Elastic Stack experience and future access to subscriptions via the Azure Marketplace. Users planning to migrate existing deployments have access to comprehensive documentation and webinars for assistance.

Nine Publishing, a prominent Australian news corporation with a 185-year history, undertook a significant overhaul of its infrastructure to maintain a competitive edge in the digital age, characterized by challenges from numerous online content creators. By adopting the Elastic Stack, Nine reimagined its content management and business insight systems, focusing on search solutions, centralized application search, logging, and analytics. Using Elasticsearch, Beats, Logstash, and Kibana, the company enhanced its ability to provide quick and relevant search results, offer content performance analytics, and improve security and reporting. The new system, led by principal systems engineer Michael Lorant, embraced a startup mentality, allowing for flexibility, agility in daily releases, and reduced costs. It incorporated best practices for node design and maintenance, such as using Logstash to create small, manageable data pipelines, enabling efficient data processing and system optimization.

Kibana's development update from November 22, 2019, highlights ongoing efforts to stabilize cloud testing, enhance security features, and migrate plugins and APIs to the New Platform. The team focused on addressing cloud test failures, expanding SaaS configurations on Azure and GCP, and improving alerting by supporting saved-object supertypes. Platform advancements include the introduction of Licensing APIs and config field whitelisting for client-side exposure, alongside UI updates supporting various frameworks. Telemetry saw new endpoint deployments and progress toward full migration under the UsageCollection plugin. Reporting efforts included transitioning export types to TypeScript and preparing Telemetry opt-in features for version 7.5. Alerting development involved activity log enhancements, alert tagging, and task management improvements, while Kibana App focused on integrating Elastic Charts and updating Kibana to TypeScript 3.7. The Maps team provided a tool for ingesting geospatial data into Elasticsearch, and design initiatives aimed at refining alerting wireframes and dashboard usability, alongside evaluating security's Role Mapping feature.

Each year, Elastic hosts the Engineering All Hands (EAH) event, a gathering of distributed team members from around the world, which this year took place in Toronto, Canada. EAH is not just an engineering event but a collaborative platform where various departments such as recruiting, marketing, and sales come together to discuss processes, engage in cross-team interactions, and strategize on product messaging and development. The event includes scheduled breakout sessions and cross-team functions to foster collaboration and drive projects that require significant time and participation across time zones. EAH also emphasizes the importance of face-to-face interaction, offering opportunities for team bonding and cultural exchange through shared meals, fun activities, and informal gatherings. Additionally, the event features the creation of video interviews to share engineers' experiences, highlighting the inclusive and collaborative culture at Elastic, which reflects the company’s commitment to building products that users love while promoting teamwork and innovation across all teams.

In the detailed exploration of Elasticsearch's authentication realms, the text delves into how these realms validate client credentials, a fundamental step before making authorization decisions within Elasticsearch's security framework. Realms, which vary by the type of credential validation method, ensure that every client request is authenticated on the coordinating node, assigning a username to the security context of the request. The authentication process is distinct from authorization, which involves assigning privileges, though some realms may incorporate role names during authentication. Supported realms include Reserved, Native, File, LDAP, Active Directory, PKI, Kerberos, SAML, and OpenID Connect, with options to implement custom realms. Credential validation varies by realm type, with some relying on HTTP headers, others on external services, and some on mutual TLS authentication. Configuration of these realms occurs in the elasticsearch.yml file, requiring system administrator privileges for changes, and caching mechanisms exist to improve performance by storing positive authentication results. The text emphasizes that, while the technical details may be complex, only a few realm types are typically used in real-world scenarios, and further resources are available for those seeking practical knowledge or troubleshooting assistance.

Ingesting geospatial data into Elasticsearch using the Geospatial Data Abstraction Library (GDAL) offers a streamlined approach for integrating various geospatial file formats, such as shapefiles, into the Elasticsearch environment. GDAL, which can be installed through package managers or compiled from source, supports over 75 different geospatial formats and requires version 3.1 or later for compatibility with Elasticsearch 7.x. Users can connect to Elasticsearch via the command line using GDAL's ogrinfo tool and secure connections with stored credentials. While ingesting data, default settings may need adjustments, such as mapping text fields as "keyword" rather than "text" to enhance search capabilities. GDAL can generate and customize mapping files for Elasticsearch, and users are advised to modify mapping parameters for compatibility with earlier Elasticsearch versions. Additional resources and forums are available for troubleshooting and expanding knowledge on GDAL command line options, making it a versatile tool for geospatial data management in Elasticsearch.

Elastic is enhancing its support for Kubernetes through advancements in observability and orchestration, focusing on addressing key challenges such as data silos, visibility in hybrid environments, and traditional pricing models. By integrating tools like Prometheus and Jaeger with Elastic Observability for Kubernetes, Elastic is providing a unified and flexible data layer for logs, metrics, and traces, optimized for containerized environments. The Elastic Cloud on Kubernetes (ECK) offers a streamlined approach to deploying and managing Elasticsearch, supporting seamless orchestration and full-lifecycle control. Elastic's commitment to innovation is further demonstrated by its involvement with the Cloud Native Computing Foundation and its continued development of integrations and tools that facilitate effective monitoring and security across cloud-native applications, ensuring users can efficiently manage and secure their Kubernetes deployments while maintaining control over their observability and security operations.

In this third installment of the "Elastic SIEM for home and small business" series, the focus is on setting up an ingest pipeline to enrich data with GeoIP information and reviewing common Beats configurations. The blog outlines the process of signing into the Kibana instance of an Elasticsearch Service deployment using an elastic superuser account to configure the necessary settings. It emphasizes the importance of standardizing data settings across various Beats such as Auditbeat, Filebeat, Packetbeat, and Winlogbeat, and provides guidance on configuring top-level processor settings and the Elastic Cloud output. Additionally, the post discusses the importance of enabling monitoring for individual Beats in Stack Monitoring in Kibana, and configuring internal queue settings for retaining events during internet outages or system downtime. Readers are encouraged to delve into documentation to tailor configurations to their specific needs, with an emphasis on preparing for data collection in the subsequent part of the series.

Logstash uses in-memory queues for buffering events between pipeline stages, but enabling persistent queues to prevent data loss can significantly impact performance, particularly due to the single-threaded nature of disk I/O. This performance drop is highlighted by a case where throughput fell by 75% when persistent queues were implemented. The slowdown occurs because a single pipeline cannot drive the disk with more than one thread, even if there are multiple inputs. To mitigate this, the blog suggests running multiple parallel Logstash pipelines within a single process and load balancing the input data across them, which can increase throughput by allowing more simultaneous disk I/O operations. In a real-world scenario, implementing four parallel pipelines improved performance significantly, bringing throughput closer to levels without persistent queues, albeit still 25% lower. This approach effectively overcomes the limitations of single-threaded disk I/O by increasing the number of concurrent threads writing to the disk.

In the second installment of the "Elastic SIEM for home and small business" series, the focus is on securing access to an Elasticsearch Service deployment by creating specific roles and user accounts to minimize the use of the elastic superuser account. The process involves setting up roles for Beats, with two main roles being beats_setup and beats_writer, designed to grant necessary access for setting up and shipping data, respectively. Separate user accounts are created for different functions, ensuring that each has the minimal privileges required for its role, thereby enhancing security. Additionally, a SIEM user role is established to provide view-only access, which allows users to monitor data without superuser privileges. The guide emphasizes the importance of using least-privilege principles and maintaining strong passwords, while also preparing for future configurations, such as enriching data with GeoIP information and installing Beats on various systems to enhance network visibility.

Outlier detection in Elastic machine learning is designed to identify abnormal data patterns across various applications, such as security analytics and fraud detection, without the necessity for a time-based analysis. Elastic's tool utilizes an ensemble of learners, combining nearest neighbor and density-based methods, to effectively detect outliers in multi-dimensional datasets. The system automatically calibrates hyperparameters and leverages ensemble learning to enhance predictive performance and scalability, making it user-friendly by not requiring manual parameter settings. Elastic's approach to outlier detection is benchmarked against publicly available datasets, showing competitive performance with state-of-the-art algorithms, despite certain limitations like downsampling that might lead to false positives in densely clustered data. The tool's performance is particularly notable in conditions with varying concentrations of outliers, though it can be outperformed by algorithms like KNN and LOF when specific parameters are manually adjusted. Elastic's outlier detection offers a streamlined solution for extracting insights from non-time-indexed data, enhancing the ability to discover actionable insights from diverse datasets.

Alex Marquardt explains how to convert local timestamps to ISO 8601 format in Elasticsearch using an ingest pipeline, which is crucial when a timestamp field is submitted without timezone information and assumed to be in UTC. By using an ingest processor, timestamps can be accurately converted if the local timezone is known, preventing display issues in applications like Kibana. The example provided demonstrates how a date processor can transform a timestamp from the Europe/Madrid timezone into the ISO 8601 format, considering daylight savings time changes. Marquardt shows how to simulate the process and verify that the pipeline correctly handles timezone offsets, including adjustments for daylight savings. Additionally, a practical example of inserting and retrieving a document within Elasticsearch is provided, emphasizing the importance of having unambiguous timestamps for accurate data representation. The blog encourages users to try this setup in their Elasticsearch cluster or test it using the free trial of Elasticsearch Service, highlighting the community forums as a resource for further questions on the topic.

This blog post provides a comprehensive guide on configuring Elasticsearch SAML authentication using Microsoft Active Directory Federation Services (ADFS) as an identity provider. It outlines the necessary prerequisites, including an Elasticsearch cluster, Kibana, and Active Directory services, and details specific configuration steps for Elasticsearch and Kibana to establish a SAML realm. The post explains how to configure the SAML realm in the elasticsearch.yml and kibana.yml files, emphasizing the importance of various parameters such as idp.metadata.path, idp.entity_id, sp.entity_id, and sp.acs. It also provides instructions on generating a metadata file for ADFS, configuring ADFS as an identity provider by setting up Relying Party Trust and claims, and creating SAML role mappings in Elasticsearch to align Active Directory groups with specific roles. The guide concludes with troubleshooting tips, suggesting reviewing Elasticsearch and ADFS logs and adjusting logging settings if authentication issues arise.

Machine learning algorithms are increasingly integral to decision-making processes in various fields, necessitating an understanding of their interpretability and the factors influencing their decisions. This is particularly important for compliance with data protection regulations like the GDPR, which require transparency in automated decision-making. Elastic has developed explanatory features in its anomaly detection products to help users identify the factors influencing anomalies detected in datasets, such as network data from corporate applications. The method involves using "influencers," or values that affect anomalies, and employs strategies such as counterfactual causation and regularization to discern potential causes of anomalies. These strategies face challenges like the ineffectiveness of pre-aggregated data, emphasizing the need for domain knowledge in configuring anomaly detection systems. Understanding these influencers is crucial for trusting machine learning systems and integrating them effectively into existing decision-making frameworks.

Elasticsearch Service has announced its compliance with the Health Insurance Portability and Accountability Act (HIPAA), allowing users to engage in a business associate agreement (BAA) for handling protected health information (PHI). This compliance enables users, such as insurance companies and medical practices that must adhere to HIPAA rules, to utilize Elasticsearch Service for processing, managing, and storing PHI across platforms like Azure, GCP, and AWS, regardless of region or subscription tier. This development is aimed at facilitating the integration of Elasticsearch's rich search capabilities with PHI data while ensuring data privacy and security. New users can explore the HIPAA-compliant Elasticsearch Service through a 14-day trial, with further details available through Elastic's sales representatives or contact page.