August 2018 Summaries

Elastic recently launched its first professional certification, the Elastic Certified Engineer, a rigorous exam designed to test a wide range of Elasticsearch skills across operations and development. Hiroshi Yoshioka from Acroquest Technology became the first person to achieve this certification, reflecting his expertise and commitment to providing top-tier Elasticsearch solutions. Yoshioka, a Senior Consultant specializing in data analytics, emphasized the importance of the certification in offering security to customers considering Elasticsearch for big data analysis and visualization. He prepared for the exam through Elastic webinars, official documentation, and comprehensive training courses led by experienced Elastic Community Engineers. The exam focuses on practical problem-solving with Elasticsearch functions, rather than mere theoretical knowledge, and is strictly monitored to ensure integrity. Yoshioka encourages other engineers to pursue the certification, highlighting its value in the IT industry and its role in enhancing one's problem-solving abilities and professional credentials.

Machine learning models, such as the Ember benchmark model used for malware classification, can experience a decline in predictive performance over time, a phenomenon known as model degradation. This issue is particularly significant in information security due to the constantly evolving nature of malware. To assess the extent of model degradation, the Ember model was trained on datasets from different months in 2017, and its performance was evaluated using the area under the ROC curve (AUC) scores on subsequent test sets. Results showed that model performance decreased when tested on data that was further in time from its training data, underscoring the need for regular retraining to maintain accuracy. Although the size of the training sets was normalized to ensure consistency, the degradation pattern persisted, suggesting it is not solely dependent on dataset size. Understanding model degradation helps in deciding optimal retraining intervals for ML models, not only in malware detection but across various applications, emphasizing its importance for researchers and practitioners in maintaining model reliability.

OTTO Motors, a division of Clearpath Robotics, utilizes the Elastic Stack to enhance its fleet of self-driving industrial vehicles operating across more than 40 countries. The company's objective is to automate tasks traditionally performed by humans, including those deemed hazardous, by leveraging robotics and IoT technologies. OTTO Motors selected the Elastic Stack for its ability to seamlessly integrate with existing software, offer easy data ingestion through HTTP/JSON APIs, and provide intuitive client libraries and data collection options. This choice allowed OTTO Motors to efficiently manage the substantial data generated by their robots, which includes over 5,000 data points every 10 seconds per robot, and to create custom dashboards for insightful data analysis. The Elastic Stack's flexibility enabled OTTO Motors to extend its use beyond IoT applications to software development QA, IT security, and dashboard creation, ultimately facilitating high-speed simulations and comprehensive fleet management.

Elastic has announced the general availability of its Real User Monitoring (RUM) JavaScript agent, which is part of its open-source Application Performance Monitoring (APM) solution. This RUM JS agent allows developers to gain insights into the end-user experience by monitoring client-side performance metrics such as "Time to First Byte" and domInteractive. Unlike backend agents, the RUM agent focuses on real user interactions within frontend applications and is designed to work with any JavaScript framework. Developers can report monitoring data to an APM Server and access it through Kibana for a comprehensive view of application performance. Elastic is also planning future enhancements like distributed tracing and framework-specific information for popular frontend frameworks. The development and feature planning for this open-source agent are conducted on platforms like their Discuss forum and Github repository, where users can provide feedback and suggest new features.

In 2018, Elastic introduced a new Azure monitoring module within Logstash 6.4 to enhance monitoring capabilities for Microsoft Azure deployments using the Elastic Stack. This module provides comprehensive insights into user activity, infrastructure changes, and Azure SQL Database health through a range of Kibana dashboards, enabling users to detect and investigate issues effectively. It integrates with Azure Monitor to centrally collect and analyze activity logs and SQL DB diagnostic logs via Azure Event Hubs, facilitating real-time analytics. The module is initially offered as a free experimental feature under the Elastic Basic License, with options to upgrade for additional functionalities like security controls and machine learning. This collaboration between Elastic and Microsoft aims to simplify the process of monitoring Azure cloud environments, offering a robust solution for operational excellence.

In the blog post, Kiju Kim and Jim Ferenczi introduce Nori, an official Elasticsearch plugin for Korean language analysis, which evolved from a Lucene module initially borrowed from the Japanese morphological analyzer. Nori efficiently segments Korean text by transforming the mecab-ko-dic dictionary into a compressed binary format, optimizing the structure for fast lookups and reducing the original size significantly. The plugin uses the Viterbi algorithm to determine the most likely segmentation path by computing transition costs dynamically, thus enhancing throughput. Benchmarks comparing Nori to other plugins like Seunjeon and Arirang show that Nori offers superior indexing throughput, handling over 3000 documents per second without significant failures. The article underscores the ongoing effort to improve language support in Lucene and Elasticsearch, encouraging users to explore Nori through the latest Elasticsearch documentation and releases.

SpeedGrapher is a computer vision tool developed to detect macro-enabled document-based phishing attacks, a prevalent form of phishing where victims are tricked into opening malware-embedded documents. Unlike Blazar, which targets homoglyph attacks, SpeedGrapher focuses on identifying malicious documents by analyzing visual cues such as prominent colors, blur detection, blank detection, optical character recognition, and icon detection using technologies like K-means clustering, YOLOv3, and OCR. The tool generates feature vectors from these analyses, which are then used to train a Random Forest classifier for predicting the likelihood of phishing in new samples. The initial model for SpeedGrapher demonstrates a high level of accuracy, with a respectable area under the ROC curve of 0.98, and highlights the potential of computer vision in enhancing security measures against evolving phishing tactics. As the development continues, the tool aims to incorporate additional features and file types to better protect users from sophisticated phishing threats.

Elastic APM 6.4.0 introduces several enhancements aimed at improving flexibility, search capabilities, and automation in application performance monitoring. The release supports Logstash and Apache Kafka as outputs for APM Server, allowing more versatile infrastructure configurations and enabling additional data processing before reaching Elasticsearch. The Kibana APM UI now features a powerful search bar for filtering transactions and errors in real-time, leveraging Elasticsearch's capabilities. Additionally, machine learning integration has been added to detect anomalies in service response times, simplifying the process of setting up alerts. The update also includes the promotion of Real User Monitoring (JavaScript) and Ruby agents to general availability, with new instrumentation added for Python and Node.js, as well as the recent promotion of Java and Go agents to beta. Looking ahead, Elastic is working on distributed tracing support to help troubleshoot latency issues in microservice architectures, inviting user feedback and community engagement through various channels.

Elastic Stack 6.4.0 introduces a range of new features and enhancements across its components, aiming to improve flexibility, efficiency, and user experience. The update includes Elastic APM enhancements like support for Logstash and Apache Kafka outputs, an integrated machine learning feature for anomaly detection, and the general availability of several APM agents. Elasticsearch adds support for Kerberos authentication, FIPS 140-2 compliance, a reloadable keystore, and a more efficient Korean language analyzer. Kibana offers a one-click sample data feature, a redesigned spy panel, and improved machine learning job management. Logstash introduces an Azure module for cloud monitoring, a new file input read mode, and a faster HTTP input plugin. Beats expands with new modules and processors, while ES-Hadoop sees the general availability of an error handler API and support for secure settings. These enhancements collectively aim to increase the power and usability of the Elastic Stack for data analysis and monitoring.

Elasticsearch 6.4.0 introduces a host of new features and improvements, emphasizing enhanced security, text processing, and client capabilities. Key security enhancements include support for Kerberos authentication, FIPS 140-2 compliance, and automatic plugin signature verification, which streamline operations in regulated environments and improve plugin integrity checks. Text processing sees advancements with the introduction of a super-fast Korean analyzer called "Nori" and the new index_phrases option for efficient phrase searches. Additionally, Elasticsearch SQL functionality is enhanced with new text manipulation functions and improved JDBC driver capabilities. The Java high-level REST client continues to expand, offering more APIs, and Painless scripting now includes better context documentation. Moreover, the X-Opaque-Id header feature is extended to audit logs, improving task tracking and cancellation. The release also improves the rollup feature, addressing bugs and expanding query capabilities, encouraging users to explore the updates and provide feedback.

Kibana 6.4.0 introduces several enhancements designed to improve user experience and functionality, including a one-click sample data feature that allows users to quickly explore Kibana's capabilities without manual data loading. The update also includes a revamped data inspection workflow, new beta tutorials for integrating with services using Metricbeat modules, and a refreshed design for managing saved objects, which now offers an improved workflow for importing and exporting index patterns and a new relationship view. Additionally, users can preview scripted fields for testing prior to implementation and create custom rules to fine-tune machine learning results. Enhancements to the APM UI include integration with machine learning for identifying response time anomalies and a new query bar for precise data searching and filtering. The release also features an Elastic Maps Service landing page for geospatial visualization and the ability to apply Vega filters to dashboards, alongside experimental API documentation for role management, Logstash configuration management, and saved objects.

The blog post provides an in-depth examination of Elasticsearch's audit trail functionality and the various configuration options available to manage it. It explains that the audit trail records client interactions with the Elasticsearch cluster, ensuring accountability and compliance by detailing actions performed on system resources, such as indexed documents and metadata. The post highlights that audit logs are crucial for tracking all actions by agents, including both system services and human users, but warns of potential performance impacts due to the verbose nature of audit logging. Two main configuration options are discussed: the ability to selectively include or exclude specific classes of events, and the use of ignore policies that filter audit records based on their attributes, thus enabling administrators to fine-tune the verbosity of logs. The document emphasizes the importance of carefully setting these options to avoid accountability gaps and suggests that the dynamic nature of these settings allows for iterative auditing improvements.

The blog post discusses the enhanced capabilities of the Elasticsearch Service on Elastic Cloud, particularly focusing on the new hot-warm architecture deployment templates that optimize logging use cases by separating data into "hot" and "warm" nodes for efficient data handling and cost-effective long-term retention. Users can deploy this architecture quickly, benefiting from Elastic Cloud's unique index curation policies, which automatically manage data movement between nodes. The service also includes features such as machine learning and security, offering a comprehensive solution for log monitoring and analysis. The post encourages users to explore these features through a 14-day free trial and provides guidance on setting up and deploying the architecture, including installing Beats for data transmission and utilizing Kibana for data visualization. The inclusion of a complimentary machine learning node further enhances the service's capabilities, allowing users to create machine learning jobs to gain deeper insights from their logs.

The text discusses the persistent threat of phishing attacks, highlighting a recent case where the Russian military spoofed websites of U.S. Senate and political organizations to steal credentials or distribute malware. This underscores the ongoing challenge of phishing, which has evolved in sophistication and remains a favored tactic for both financially motivated criminals and nation-state actors. The text introduces Blazar, a computer vision-based tool developed by Endgame Research, designed to detect homoglyph attacks by identifying visually similar URLs that spoof legitimate domains. Blazar employs a Siamese neural network to transform text into images and compare feature vectors for potential phishing attempts. The text further explains the benefits of using computer vision in cybersecurity and mentions another project, SpeedGrapher, which focuses on detecting MS Word macro malware. These tools, when combined with traditional phishing detection methods, offer a robust defense against increasingly complex phishing schemes.

The text provides an in-depth explanation of the experimental Ranking Evaluation API in Elasticsearch, which is designed to evaluate and enhance the quality of search results. This API supports developers in creating and refining search queries by allowing them to measure search performance using real-life data, such as documents from Wikipedia. The text walks through setting up a demo project with Elasticsearch and highlights the process of developing and improving search queries using a dataset from Wikimedia's Discernatron, a tool for gathering human judgments of search relevance. It details how the API can be used to assess ranking quality and discusses the implementation of evaluation metrics like Precision and Discounted Cumulative Gain to guide query optimization. The API is positioned as a valuable tool for maintaining search quality, enabling structured evaluations and iterative improvements in a search system.

In the Keeping up with Kibana update, Raya Fratkina outlines the latest developments and openings within the Kibana project as of August 2018. The team is hiring for several positions, including Senior JavaScript Engineer and Software Engineer in Test. Kibana 6.4 has been released, with efforts focused on enhancing security, particularly around Spaces, and developing a new user interface for managing related privileges. The team is also addressing accessibility issues identified in version 6.3, aiming for resolution by version 6.5. Canvas features and server-side code parsing are under development, with improvements in Elastic Maps Service and data production, while several visualization pull requests are being worked on to enhance functionality and fix issues.

Elasticsearch Service on Elastic Cloud has introduced new features that enhance flexibility and customization in cloud deployments, allowing users to better align their workloads with specific hardware profiles. These profiles—optimized for I/O, compute, memory, or a hot-warm configuration—enable tailored deployments that cater to varied use cases, such as CPU-intensive tasks or memory-heavy operations. Users can further customize deployments by configuring hardware for different nodes and instances, optimizing performance and cost-efficiency. Additionally, these enhancements facilitate more precise routing of requests to suitable nodes, improving overall system efficiency. The new functionalities are also slated for inclusion in Elastic Cloud Enterprise, enabling cloud administrators to tag and configure hardware for specific use cases, thereby streamlining infrastructure management and deployment creation through templates. These developments aim to provide a more optimized, cost-effective, and adaptable experience for Elasticsearch users.

Engineers looking to enhance the relevance of search results in Elasticsearch applications can leverage the Ranking Evaluation API to achieve better outcomes. The blog post emphasizes the importance of tuning search relevance, which involves optimizing combinations of data, data models, and query templates. Relevance tuning is inherently subjective and varies based on unique datasets and user expectations, making test-driven approaches crucial. The Ranking Evaluation API, introduced in Elasticsearch version 6.2, allows engineers to measure search quality using information retrieval metrics like precision and mean reciprocal rank. To utilize this tool, engineers must gather a representative sample of queries and relevance judgments for evaluation. The process involves iteratively adjusting queries and configurations to improve search quality, ensuring that the system meets quality standards. The post also advises periodic re-evaluation as changes in data and user behavior can influence relevancy, and encourages feedback on the process.

DNS tunneling is a technique that exploits the Domain Name System (DNS) to transmit other protocols' data through DNS queries and responses, often used for command and control or data exfiltration by malware. Despite advancements in internet architecture and detection techniques, DNS tunneling remains challenging to detect due to its ubiquity and the false positives generated by legitimate services like Content Delivery Networks (CDNs) and unconventional DNS applications. The article highlights that DNS tunneling can achieve covert communication without direct connections to attackers and can be difficult to detect due to its stealth and performance characteristics. Detection requires a layered approach, considering record types, packet sizes, and access patterns, while noting that future DNS privacy protocols like DNS over HTTPS and DNS over TLS may further complicate detection efforts.

In 2018, Lyft transitioned its operational logging system from Splunk Cloud to Amazon Elasticsearch Service (Amazon ES) and eventually to a self-managed Elasticsearch deployment, seeking to unify observability and security data while reducing costs and administrative overhead. Initially, Lyft's Observability team faced challenges with Splunk's retention limits, ingest backups, and scaling costs, prompting a switch to Amazon ES. However, limitations such as outdated versions, storage performance issues, and lack of direct cluster access led to operational inefficiencies, prompting another migration. With a slightly larger team than that used for the Splunk migration, Lyft successfully moved to a self-managed Elasticsearch in two weeks, gaining full control over system features and operational aspects. This transition allowed Lyft to handle increased log ingestion rates and to resolve issues independently, improving the overall reliability and efficiency of its logging infrastructure.

Elasticsearch's machine learning features, aimed at detecting anomalies in time series data, support various use cases such as identifying suspicious activities and planning routes. However, configuring optimal machine learning jobs can be complex due to the multitude of features available. To address this, Elasticsearch introduced automated job validation in Elastic Stack 6.3, enhanced in version 6.4, which provides detailed feedback on job configuration and links to relevant documentation. This feature, accessible on the jobs list page and within job creation wizards, helps users refine their settings by analyzing the configuration and underlying data, offering suggestions for improvements. Key aspects of job validation include checks on aggregatable fields, bucket span, time range, cardinality, and influencers, ensuring that users can preemptively address potential issues and optimize job configurations. An example illustrates how job validation spotted a typo in a configuration, leading to a series of adjustments that resulted in a successful validation, demonstrating how this tool aids in creating more effective analysis setups.

Samtec's Smart Platform Group, initially focused on high-speed copper and optical interconnects, realized the potential of Elasticsearch and Kibana for business intelligence and analytics beyond their typical use cases in logging and security. Starting from a need to manage and analyze vast amounts of log data from manufacturing processes, Samtec transitioned to using these tools for broader business intelligence applications, replacing tools like Tableau with Kibana dashboards for real-time order status displays and using machine learning for anomaly detection. They developed Conveyor, an open-source plugin designed to simplify data imports into Elasticsearch, enabling self-service business intelligence by allowing data analysts and business users to easily integrate various data sources. This transformation has empowered them to conduct comprehensive analyses, such as tracing suspect lots through manufacturing processes and integrating diverse data sources for powerful business insights. Samtec continues to explore and share advancements with the Elastic Stack, underscoring its belief in its capacity as a robust open-source business intelligence platform.

Elasticsearch Service on Elastic Cloud provides a flexible and efficient solution for handling logging and metrics workloads, offering various hardware choices and deployment templates. Users can opt for different architecture types such as uniform or hot-warm, with each having distinct features and suitability based on storage speed and use-case requirements. The hot-warm architecture uses 'hot' nodes for recent, frequently accessed data with fast SSD storage, and 'warm' nodes for long-term storage using slower, cost-effective storage options. Efficient data management involves optimizing index mappings, maintaining large shard sizes, and compressing JSON data to reduce disk usage. The service is available on AWS, GCP, and now Azure, with node configurations tailored for performance and storage needs. Sizing an Elasticsearch cluster involves estimating data volume, considering storage and query requirements, and choosing appropriate node types, with recommendations for master and coordinating nodes to enhance resilience and performance. Elastic Cloud facilitates easy setup and management, offering a free trial for users to explore its capabilities.

The guide outlines the necessary changes developers must make to update their Kibana plugins for compatibility with version 6.4, emphasizing improved ease of plugin development. Key updates include replacing the "uses" array with explicit imports, modifying the visualization loader to directly accept filters and queries, and introducing the Inspector to replace spy panels. Developers working with editor states are advised to adjust their code depending on whether they're using Angular or React, with specific instructions for managing editor parameters. Angular dependencies are being reduced, with services like RegistryFieldFormatsProvider and the Timefilter Angular service undergoing significant changes, including the transition to singletons and the introduction of new event handlers. These updates aim to streamline the plugin development process and enhance compatibility with newer Kibana versions.

"Keeping up with Kibana" is a series that highlights recent developments and updates in the Kibana project, including hiring opportunities for various engineering roles. The latest updates cover several areas such as security, where the team is focusing on Spaces and its related user experience, as well as ongoing efforts to secure it. Design improvements continue with the introduction of new components and TypeScript support in EUI, while the platform team works on UI enhancements targeting future releases. The Canvas team is enhancing functionality with new features like drag and drop pages and PDF reporting. The management team has shifted focus back to index lifecycle management and rollup support, while the operations team progresses with DLL vendor work for client dependencies. Additionally, there are updates in various PRs addressing fixes and improvements across different functionalities within Kibana.

Anonymize-it is a tool developed by the Elastic Machine Learning team to address data privacy concerns by facilitating pseudonymization, which is crucial when dealing with sensitive data that cannot be freely shared due to privacy regulations. The tool is designed to help users suppress, mask, or generalize personal identifiers and quasi-identifiers in datasets while preserving the data's behavioral characteristics, enabling the safe sharing of information for machine learning and analytics purposes. It includes components for reading data from sources like Elasticsearch, anonymizing it using Python's Faker package, and writing the anonymized data to destinations such as a local filesystem or Google Cloud Storage. While the tool is not intended to meet GDPR anonymization requirements, it assists in pseudonymization by replacing real data values with artificial ones that maintain the original data's semantics. Anonymize-it highlights the importance of data privacy in today's digital landscape, offering a practical solution for organizations needing to protect sensitive information while leveraging data for development and analysis.

Elasticsearch Service on Elastic Cloud has introduced a new pricing model aimed at reducing costs and increasing transparency and flexibility for its users. Starting at $16.40 per month, the revised pricing is significantly lower than the previous $45, making it easier for users to adopt the service. The new model separates charges for data transfer, snapshot storage, and deployment costs, offering a more straightforward, linear pricing structure that scales with the size and number of availability zones. A new free allowance includes 1 GB Kibana and machine learning instances, alongside a certain amount of free snapshot storage and data transfer, to help users start without incurring extra costs. Existing Standard monthly customers will automatically benefit from the new reduced pricing, while annual customers will continue to receive credits based on their agreements. The changes also provide more transparency regarding the underlying cloud resources used in deployments, allowing users to choose specific instance configurations on AWS and Google Cloud Platform.