December 2018 Summaries

The Elastic Advent Calendar 2018 concluded with a diverse array of 25 articles that explored various features and applications of the Elastic Stack, including Kibana, Elasticsearch, Logstash, and more. These articles, written in multiple languages such as French, Japanese, Korean, and Chinese, covered topics like enriching postal addresses using Open Street Map data, understanding search analyzers in Kibana, implementing typeahead search with Elastic App Search, conducting rarity analysis with machine learning, and managing search relevancy. The series also delved into advanced features like Chaining Ingest Pipelines, using SQL for querying Elasticsearch data, and Cross Cluster Replication. Furthermore, the series highlighted the practical applications of these technologies in areas such as security analytics, observability, and Python integration. The Elastic community was encouraged to engage with these topics on the Elastic Discuss Forums, ensuring that the knowledge shared continues to inspire innovative uses of the Elastic Stack.

Functionbeat is a serverless data ingestion tool that integrates with Elasticsearch, providing a streamlined approach to ingesting cloud data from services like AWS Lambda. It functions as part of the Beats product suite, allowing users to deploy it on serverless compute platforms to efficiently process and ship events such as logs and metrics from cloud infrastructure to Elasticsearch. Initially supporting AWS Cloudwatch Logs and SQS, Functionbeat is designed to expand its compatibility with more cloud providers and data sources over time. Users can configure Functionbeat through a user-friendly interface without needing to write code, enabling the transformation, enrichment, and routing of data. It operates similarly to other Beats, such as Filebeat and Metricbeat, and is freely available under the Elastic Basic license, encouraging feedback and contributions from its open-source community as it progresses towards general availability.

CVE-2018-17246 is a Local File Inclusion (LFI) bug in Kibana that allows a remote attacker to load and execute files from the local filesystem, potentially causing security vulnerabilities. This flaw, which was publicized through various tweets and news stories, was addressed in Kibana versions 6.4.3 and 5.6.13, with updates promptly released by Elastic on November 6, following an alert from Cyberark on October 23. While the vulnerability could theoretically be used to crash Kibana, it does not allow attackers to view the contents of arbitrary files. Elastic recommends applying security updates to mitigate the issue, but a temporary workaround involves disabling the Kibana Console plugin. Elastic emphasizes the importance of timely updates and encourages reporting potential security flaws to their security team to ensure swift resolutions.

Crimson Macaw collaborated with Manchester Airport Group to develop real-time dashboards for Stansted Airport's security operations using Canvas and Elasticsearch, aiming to enhance the control room and security staff's decision-making capabilities by providing a comprehensive view of passenger flow and security performance. The project involved overcoming challenges related to data ingestion from various sources, such as on-premises databases and external APIs, and required the creation of custom plugins for data visualization and control, including handling gzip compressed messages and formatting timestamps with time zones. Canvas, a new visualisation tool discovered during an AWS Summit, enabled more granular control over dashboard elements than Kibana, prompting the development of additional features like colour coding and dynamic image updates through plugins. The iterative design process, involving stakeholder feedback and workshops, led to the creation of visually appealing and functional dashboards, such as the Security Hub and Train Arrivals dashboards, which display real-time data to facilitate quick decision-making by airport security staff. The collaboration culminated in a modern display of integrated data that improved internal efficiencies and demonstrated the potential of Canvas for creating imaginative and effective data visualisations.

Version 6.5.4 of the Elastic Stack, released on December 19, 2018, addresses several key issues in Beats and Kibana, including a fix for a regression in Kibana where the Elasticsearch timeout setting of 0 is now correctly interpreted as "no timeout," preventing visualizations from returning partial data or errors. Additionally, the "Show partial buckets" option has been reinstated in the Kibana Visualize app, and a Go routine memory leak fix in Filebeat has been incorporated into this version. The update also addresses a potential CPU denial of service vulnerability in certificate validation, identified as CVE-2018-16875. Users are encouraged to upgrade to this version for improved stability and security, with further details available in the release notes for Elasticsearch, Kibana, Beats, and Logstash.

Elastic App Search, built on top of Elasticsearch, offers a user-friendly solution for implementing advanced search functionalities such as autocomplete, faceting, and tune-able relevance with a concise API and an intuitive admin dashboard accessible to both developers and non-developers. Despite exposing fewer APIs compared to Elasticsearch, App Search supports complex data models and relevance scoring requirements through features like Result Grouping and Boosting. Result Grouping helps streamline search results by aggregating them at a higher level, enhancing user experience by preventing overwhelming displays of similar items. Boosting, including Value, Functional, and Proximity Boosts, allows fine-tuning of search result scores based on various field values and functions, providing control over how specific attributes influence search rankings. These features are configurable via the Admin Dashboard, making it possible for non-developers to adjust search parameters without modifying application code, offering flexibility and ease of integration for a range of search use cases.

Elasticsearch's ingest node, a component used for pre-indexing document processing, has been enhanced in version 6.5 with new capabilities, including conditional execution of processors using the Painless scripting language. This update allows processors to be skipped or executed based on specific conditions, such as renaming fields only if certain criteria are met, and enables the use of a drop processor to prevent indexing of unwanted documents. Additionally, the introduction of the pipeline processor facilitates the reuse and organization of multiple pipelines, while the dissect processor offers a simpler, potentially faster alternative to the Grok processor for parsing log lines. Other improvements in this release include the ability to set default pipelines for indexes, enhanced documentation, improved node-level statistics, updates to Kibana Console's auto-complete functionality, and a new Rally challenge for benchmarking Grok performance.

The Kibana weekly update for December 14, 2018, highlights significant changes and developments, including the removal of PhantomJS in favor of Chromium as the sole option for Reporting's headless browser in version 7.0. The update lists job openings for various senior and engineering positions, discusses roadmap themes such as alerting service implementation and GIS app improvements, and details ongoing work for the Kibana 6.6 version. Notable enhancements include the introduction of trigonometry functions in TinyMath, Plotly experiments for rendering charts in Canvas, and updates to EUI's design palette. Additionally, the update provides insights into platform improvements, such as the development of a new saved objects import/export API, and various pull requests addressing both design and operational aspects.

Elastic Cloud Enterprise (ECE) 2.0 introduces new capabilities that enhance the flexibility and scalability of managing Elastic Stack deployments, offering features such as allocator tags, instance configurations, and deployment templates. These tools help optimize resource utilization by allowing users to match specific components to appropriate hardware based on use cases like logging, which can benefit from a hot-warm architecture. The system also enables different configurations for components like Kibana, master nodes, and ingest nodes, prioritizing memory or CPU as needed. The platform supports the separation of environments into development, staging, and production, each with different high availability and network requirements. ECE 2.0 also allows for the isolation of deployments based on team needs, facilitating better management of resources and adherence to varying service level agreements. These features are designed to improve performance, offer isolation between environments and teams, and provide a customizable Elastic experience.

Beats Central Management, introduced in version 6.5 of the Elastic Stack, offers a centralized Kibana UI for managing and updating Beats configurations, aiming to reduce reliance on external tools like Ansible, Puppet, or Chef. This feature allows users to manage Beats configurations securely and efficiently, even at scale, with the help of tags for organized and flexible configuration management. It is available under the Elastic Gold license or Standard license when using the Elastic Cloud service, ensuring a secure deployment. Key security measures include unique authentication tokens, additional user roles, and auto-expiring enrollment tokens. Although currently in beta, this feature is designed to provide users with a seamless management experience, and feedback is actively encouraged to improve and expand its capabilities.

The article by Alex Marquardt provides a detailed guide on configuring security for Elasticsearch, particularly focusing on enabling TLS/SSL encryption and PKI authentication. It emphasizes that as of Elasticsearch versions 6.8/7.1, security features including TLS encryption, user authentication, and role-based access control can be enabled for free. The guide outlines the process of setting up TLS/SSL for both transport and HTTP communications within an Elasticsearch cluster, noting the importance of using appropriate certificates for secure data transmission. For organizations that prohibit storing credentials in configuration files, the article suggests using Public Key Infrastructure (PKI) client certificates as an alternative authentication method. Additionally, it explains how to configure Kibana to authenticate with Elasticsearch using these methods, initially through a username and password, and subsequently through a client certificate approach. The guide provides step-by-step instructions, including generating certificates, modifying configuration files, and testing authentication to ensure secure communication and operation within the Elasticsearch environment.

Imec, a leading nano-electronics research institute, has adopted a novel approach to analyzing its high-tech manufacturing supply chain by employing social network theory through the Elastic Stack framework. This method involves modeling the business network as a social network graph, where nodes represent customers, suppliers, and projects, with connections symbolizing financial transactions. The analysis revealed a denser network structure over time, driven by an increase in suppliers offering more complex services and the emergence of "super-customers" who frequently engage in multiple projects. By generating numerous business partner network graphs spanning from 2004 to the present and analyzing them in MATLAB, Imec has been able to track and understand significant changes in its business model, illustrating the effectiveness of using social network theory to study business network evolution.

The blog post by Alex Marquardt addresses the issue of duplicate documents in Elasticsearch, particularly when auto-generated IDs are used, leading to the same document being stored multiple times with different IDs. To tackle this, the post outlines two methods for detecting and removing duplicates. The first method utilizes Logstash, employing a fingerprint filter to generate unique IDs based on specific field values, thereby preventing duplicates by ensuring documents with identical content share the same ID. The second method involves a custom Python script that calculates hashes for specified fields and uses these hashes to identify duplicates by storing document IDs in a dictionary. This approach is memory-efficient and can be enhanced to handle time-series data more effectively by processing subsets of documents within specific time frames. The blog also considers potential issues like hash collisions and provides insights into optimizing deduplication processes for Elasticsearch users.

Version 6.5.3 of the Elastic Stack was released on December 11, 2018, addressing notable issues in Beats and Kibana. Key bug fixes include the restoration of the "group by terms" functionality in Kibana's Time Series Visual Builder, which was affected by CSS styling changes introduced in version 6.5, and the correction of rfc3164 syslog parsing for December in Filebeat, which was due to a typo in the log parsing code. Users are advised to upgrade to this latest version for improved performance, and further details can be found in the release notes for Kibana, Elasticsearch, and Beats, noting that there are no changes in Logstash.

Elastic is enhancing its cloud-native capabilities by releasing Helm charts for Elasticsearch and Kibana, and joining the Cloud Native Computing Foundation (CNCF). These developments aim to simplify the deployment of Elastic Stack on Kubernetes, leveraging Helm as a package manager to streamline the process. The Elastic Stack, including data collectors like Filebeat and Metricbeat, offers comprehensive observability tools that integrate with Kubernetes to monitor logs, metrics, and tracing data. Elastic APM supports distributed tracing, allowing for detailed performance analysis of microservices architectures. Joining CNCF aligns with Elastic's commitment to open-source principles and fosters collaboration on open tracing and metrics standards within the cloud-native community. Elastic's participation in KubeCon North America 2018 highlights these advancements and showcases its integration capabilities with Kubernetes and other CNCF projects.

The Elastic Advent Calendar 2018, inspired by previous initiatives, offered 25 engaging topics shared by the Elastic Engineering team over the first 25 days of December. Each day featured a new post in various languages, showcasing diverse aspects of the Elastic Stack technologies. Highlights from the first week included enriching postal addresses with Logstash, understanding the Analyze UI plugin for Kibana, crawling Facebook data with Logstash, applying rarity analysis through machine learning, implementing typeahead search with Elastic App Search, addressing search relevancy in Elasticsearch, and exploring Kibana Canvas introduced in Elastic 6.5. The initiative aimed to engage the community with insightful content, inviting feedback and discussion.

The weekly update for Kibana, dated December 7, 2018, highlights several key developments and ongoing projects. The team continues to progress on granular application privileges, opting for a new approach to assigning Kibana privileges alongside roles for effective cluster/index privileges. The Alerting Service is being redesigned to support pluggable actions and multiple alert types, akin to ElastAlerts, with ongoing work on defining pluggable actions. The Geo GIS App has seen improvements, including test coverage enhancements, Web Map Service support, and new join editor for choropleth mapping. Canvas now supports global CSS overrides, and there is a detailed example of using SQL with Canvas to compute percentages of delivered products. The update also mentions design changes, including TypeScript support in EUI, navigation changes, and the development of the Super Date Picker component. Reporting enhancements involve resolving issues, improving logging and error handling, and creating documentation for multiple .kibana indices configuration. Additionally, the update covers bug fixes, changes in the platform, and operations like extending pre-commit hook scripts and optimizer cache invalidation.

Since joining Elastic in 2015, George Young reflects on the evolution of the company's public sector events, highlighted by the recent Elastic{ON} Tour held at the Marriott Marquis in Washington DC. Unlike the inaugural 2015 event which was a modest gathering with 283 attendees, this year's event attracted over 1200 registrants from various government sectors, prompting a split into two equally popular tracks focusing on security/cyber and search/analytics. Technical presentations showcased how public sector entities like the Hill Enterprise Data Center and Army Research Labs utilize Elastic for monitoring, security, and policy compliance. The event also featured Elastic experts discussing advancements in their products, while providing training sessions on security analytics and Elasticsearch. Despite the event's growth, the core mission of fostering community connections and collaboration remains unchanged, with attendees expressing enthusiasm over new features and the opportunity to engage with fellow users. The event underscored a vibrant community spirit, encouraging users to share ideas and best practices, and plans are already underway for a larger venue to accommodate future demand.

Event Query Language (EQL) is a versatile tool designed to express relationships between events, allowing users to write adversarial detections that are not limited by the underlying technology. Originally integrated into the Endgame platform for behavior-based detections, EQL has been open-sourced, offering flexibility for various data analysis tasks such as searching, data stacking, and complex behavior hunting. EQL supports Python versions 2.7 and 3.5-3.7 and can be installed via PyPi. To facilitate user engagement, a static test dataset and an EQL Analytics Library, which includes behavior-based detections mapped to MITRE ATT&CK™, are provided. The guide explains how to generate data using Sysmon and Atomic Red Team for testing EQL queries. EQL allows users to analyze different event types and sequence them to enhance detection accuracy, reducing false positives. The text emphasizes the importance of data normalization and platform agnosticism in EQL queries, highlighting the ability to adapt to various data sources through schema mapping. The article encourages community contributions to expand analytics and shares insights into the analytic schema, which is structured in TOML, to detail rule creation and maintenance.

Josh Bressers explores the dynamics of Bitcoin's price behavior and mining activity using Elasticsearch and Kibana, highlighting the intricate relationship between Bitcoin's blockchain data, transaction volume, and mining difficulty. He observes that the Bitcoin network consists of over 550,000 blocks mined through complex mathematical equations, where the difficulty level adjusts to maintain a consistent block generation rate. The recent decline in Bitcoin's price from a peak of $20,000 is linked to increased transaction volume as people sell off their holdings, while a notable decrease in mining difficulty suggests fewer miners are active due to the high costs and limited utility of specialized mining equipment. Bressers emphasizes the expensive nature of Bitcoin mining, which relies on Application Specific Integrated Circuits (ASICs) that can only compute SHA-256 hashes, rendering them unusable for other purposes. The decrease in active mining rigs indicates that some miners may be retreating due to negative returns on their investments, despite the historical trend of growing difficulty and miner participation.