July 2018 Summaries

The blog post provides a practical guide on tuning the performance of Elastic Beats, specifically focusing on optimizing Filebeat's ingestion rates into Elasticsearch by adjusting batch size and worker count. The author highlights that performance tuning is an iterative process, often revealing bottlenecks not in Beats but in subsequent stages. Using a controlled experiment setup with an Elasticsearch cluster and a consistent log file, various batch sizes and worker counts were tested, demonstrating that increasing these parameters does not always yield better performance and that optimal settings depend on specific conditions. The post emphasizes the importance of methodical testing in a stable, repeatable environment and acknowledges that while tuning these parameters can significantly improve performance, it may also increase resource consumption. The article concludes with a preview of more advanced tuning options, suggesting that further tuning can lead to even higher throughput but with additional resource trade-offs.

Kellen Evan and James Rucker's tutorial outlines the process of building an application search feature using Ruby on Rails and Elastic App Search, formerly known as Swiftype App Search. The guide emphasizes the importance of a fast and efficient search experience for users, highlighting how Elastic App Search can enhance user interactions with web applications by providing powerful, intuitive search capabilities. The tutorial provides step-by-step instructions on setting up a sample search application, creating an Engine within App Search, configuring the App Search Ruby Client, and ingesting documents. It also covers schema alteration and fine-tuning search relevance. The tutorial shows how to synchronize database changes with App Search using Active Record Lifecycle Callbacks and offers guidance on using a rake task for mass data ingestion. Additionally, it demonstrates how to alter the RubyGemsController to implement a search bar and how to enhance search relevance by weighting different fields. The tutorial concludes by discussing the potential for more advanced search features, such as filtering results based on popularity, and highlights the breadth of possibilities offered by Elastic App Search for crafting engaging and efficient search experiences.

Elastic has acquired Insight.io, a company known for its modern source code search platform, to enhance its search capabilities by integrating Insight.io's semantic search and code intelligence solutions. Insight.io, which uses Elasticsearch for its core features, was founded to improve code development efficiency and provide deeper insights into software, aligning well with Elastic's mission to make search an indispensable tool across various sectors. The merger aims to combine Elastic's robust search technologies with Insight.io's innovative platform, promising improved code search capabilities that support developers in navigating complex codebases efficiently. The collaboration is expected to leverage Elastic's resources to scale Insight.io's product and deliver enhanced user experiences across several use cases, including application search, logging, metrics, and security analytics. Both companies share a vision for innovation and efficiency in software development, seeing this partnership as a natural fit to advance their goals and provide significant value to developers worldwide.

In the July 23, 2018 edition of "Keeping up with Kibana," several updates and developments were highlighted within the Kibana project. The design team embarked on a significant transition from using Less to Sass to enhance styling consistency and theming, while also aligning Kibana with the latest version of the Elastic UI (EUI). A new pattern for the K7 date selector and updates to the EuiCard and visualization type glyphs were introduced. The management team added the capability to validate scripted fields directly in the UI, alongside improvements in scripted fields preview and validation. Progress continued on index migrations, and several bugs, including a chromium bug and a sample data gap issue, were addressed. Additionally, the visualization team worked on refactoring core rendering infrastructure and loaders to TypeScript and fixed a regression issue causing flickering in visualizations during rerendering. There was also a guest blog post by Edan Shahmoon from Micro Focus discussing experiences with the Kibana Canvas feature.

The blog post provides a detailed guide on configuring SAML-based Single Sign-On (SSO) for Elasticsearch using Azure Active Directory (AAD), facilitated by the Elastic Azure Resource Manager (ARM) template. It outlines the steps required to set up an Enterprise Application in AAD, which acts as the interface between AAD and Elasticsearch, and configure it for SAML SSO. The post highlights the importance of the SAML token, which eliminates the need to seed Elasticsearch with user accounts by relying on claims within the token to determine user identity and privileges. The guide includes instructions on configuring role claims and optionally group claims, as well as setting up role mappings in Elasticsearch to assign roles to users authenticated through AAD. It also explains the necessary configurations for deploying Elasticsearch on Azure with SSL/TLS to ensure secure communications. The post concludes by emphasizing the simplicity of integrating AAD for SAML SSO with Elasticsearch and hints at future improvements with an Elasticsearch application in the Azure Active Directory gallery.

The blog post introduces Elasticsearch SQL, a new feature in the Elastic Stack 6.3 release, which allows users to query Elasticsearch using SQL syntax. This addition aims to make Elasticsearch more accessible, especially for new users and data consumers familiar with SQL, by providing a read-only interface compliant with a subset of the ANSI SQL specification. The post discusses the challenges of integrating SQL with Elasticsearch's document-oriented model and highlights the benefits of using SQL for structured analytics, contrasting it with Elasticsearch's native Domain Specific Language (DSL) for full-text search. Although SQL cannot transform Elasticsearch into a fully relational database, it leverages Elasticsearch's performance for data retrieval and manipulation. The implementation includes phases such as parsing, analyzing, optimizing, and executing queries, with support for JDBC connectivity. The post also explores the mapping of Elasticsearch concepts like indexes and documents to SQL tables and rows, and demonstrates how SQL queries can be translated into Elasticsearch DSL, offering an efficient way to utilize Elasticsearch's capabilities.

Logstash Lines announces updates and improvements in Logstash, highlighting the release of Logstash 6.3 and progress towards supporting NFS. The Logstash team has focused on enhancing the file input plugin by improving file rotation handling and remote filesystem file reading, verified by a user on NFS who experienced issues with ASCII NUL bytes. Key fixes include addressing broken classpath issues, enhancing file rotation scheme support, and optimizing remote file processing to prevent data loss. Additionally, updates in documentation and plugins, such as the logstash-input-file and logstash-output-graphite, address various technical issues and enhancements, including regression fixes, exception handling, and upgrading to Gradle 4.8.1, while also expanding plugin documentation and adding new input options like azure_event_hubs.

The continuation of the Elasticsearch SQL series delves into more advanced features of Elasticsearch's SQL capabilities, building on the initial introduction of its SQL feature and _translate API. The article explores complex functionalities like GROUP BY using Composite Aggregation, which allows scalable data grouping without memory limitations and filtering groups with the HAVING operator. It highlights the use of the QUERY and MATCH operators to leverage Elasticsearch's unique text search capabilities, allowing more nuanced searches compared to traditional RDBMS systems. It also discusses cross-index searches using aliases, illustrating how queries can span multiple indices with identical mappings. While Elasticsearch SQL currently lacks traditional JOIN support, it can handle nested documents, offering some degree of relational modeling. The article concludes by mentioning current limitations in cross-index queries and nested SELECT clauses, while hinting at future improvements such as multi-level GROUP BY, geospatial operators, and enhanced date/time functions, showcasing the ongoing evolution of Elasticsearch SQL.

Integrating Nagios checks with Logstash provides a centralized monitoring solution that complements the distributed monitoring capabilities of Beats. Users can utilize the logstash-input-nagioscheck plugin to parse the output of Nagios checks and push performance data metrics into the Logstash pipeline. This integration facilitates the storage and analysis of custom polling data, potentially allowing indefinite retention with Elasticsearch's Rollup feature and enabling trend forecasting with machine learning. The process involves installing the necessary plugins, setting up Elasticsearch and Kibana, and configuring specific checks for data ingestion. Despite being community-maintained, the integration offers a flexible and simple interface for users to write or download Nagios checks, making it accessible even for those with minimal scripting skills.

The weekly update on Kibana from July 9th, 2018, highlights several significant developments and improvements in the platform. The Elastic Maps Service now has a dedicated landing page to facilitate data integration with Elasticsearch in region maps, while enhancements to Canvas, such as automatic expression formatting and case-insensitive functions, have been introduced to improve usability. Efforts in platform localization are advancing, with key features becoming translatable, and the new platform foundation is ready on the server side, focusing on services like saved objects and Elasticsearch. Security updates include modifications to deprecation logging and integration tests, and the design team is resolving bugs and working on new prototypes. In management, progress is being made on rollup index patterns, and the visualizations team is removing Angular dependencies to prepare for rendering with the Canvas pipeline. Finally, experimental features like the EuiXYChart in EUI have been added, and various bug fixes and enhancements have been implemented across the platform.

Monitoring container environments such as those using IBM Cloud Kubernetes Service involves gathering and analyzing logs, metrics, and metadata for comprehensive insight into application performance and infrastructure management. By utilizing tools like Beats, lightweight shippers for logs and metrics, data can be collected from various levels, including orchestration, hosts, applications, and containers, with Beats deployed as a DaemonSet within the Kubernetes cluster. Elasticsearch Service is used to index, store, search, analyze, and visualize this data, while Beat modules simplify the process by providing pre-packaged dashboards and parsing capabilities for common log formats. These modules, which include specific applications like Apache and Redis, offer a powerful way to collect and visualize data, providing actionable intelligence through enriched performance charts that incorporate Kubernetes events and metadata. The blog emphasizes the importance of monitoring beyond what cloud providers manage, aiming to provide a detailed understanding of infrastructure and application performance through comprehensive data collection and analysis.

Elasticsearch's X-Pack security extensions offer the ability to develop custom security realms and roles providers, enhancing the built-in authentication and authorization capabilities for diverse authentication mechanisms. With version 6.3, these extensions are now treated as standard Elasticsearch plugins, allowing them to overcome previous limitations such as classloader issues and enabling them to implement various plugin interfaces. This change introduces the SecurityExtension interface, which simplifies the conversion of existing extensions and facilitates new development. A practical example discussed is a custom security extension designed for a hypothetical eCommerce platform, which dynamically generates roles based on store-specific data without requiring manual role definitions. This extension uses cryptographic signatures to authenticate users via custom HTTP headers, ensuring secure and efficient access control across multiple independent stores. The described security extension is illustrative, providing a comprehensive example for developers to learn from, although it is not production-ready due to the absence of tests and limited error handling.

Rally 1.0.0, a benchmarking tool specifically designed for Elasticsearch, has been released after over two years of development involving more than 1,400 commits from 27 contributors. This release includes a load generator and an optional component to configure an Elasticsearch cluster, supporting versions from 1.x to the latest 6.3 release. Rally allows users to define their own benchmark scenarios, known as "tracks," to achieve realistic results while providing basic reporting and an optional Elasticsearch metrics store for comprehensive documentation and custom visualizations. The tool's built-in telemetry offers deeper insights into cluster performance, and future plans include simplifying benchmark creation and enhancing reporting features like a dashboard generator. The release signifies a significant milestone, with ongoing ambitions to further improve the tool's capabilities for Elasticsearch performance evaluations.

Elastic announced the beta release of the Elastic APM Java agent, which is designed for testing in development environments but not yet ready for production use. This agent is part of Elastic's Application Performance Monitoring solution, offering insights into application performance and error tracking, with additional agents available for various other programming languages. The Java agent supports Spring Boot applications and a range of application servers, providing auto instrumentation without code changes while allowing for programmatic customization and integration with OpenTracing. It efficiently records and transmits HTTP traffic and JDBC queries to an APM server, with minimal overhead, for visualization in the APM UI of Kibana. Elastic is working on adding distributed tracing capabilities and expanding support for more frameworks, inviting user feedback and contributions to the project.

Keeping up with Kibana highlights the latest developments and enhancements in the Kibana project as of early July 2018. Key updates include job openings for various engineering roles, progress on internationalization with the integration of a base i18n engine for translation, and advancements in GIS capabilities with new data layers for region maps. The team is working on a new GIS UI plugin, improvements in the Management section to support rollup indices, and several PRs addressing various optimizations and refactoring tasks. The platform is transitioning to a new server infrastructure, aiming to support both legacy and new platform applications, while the visualizations team continues to enhance the chart components and editor infrastructure. Design efforts focus on refining the popover service and developing responsive layouts for the new visualization editor. Updates to documentation and EUI versions are ongoing, alongside security improvements with the introduction of Role Based Access Control (RBAC) and functional test framework migrations.