September 2017 Summaries

In September 2017, Elastic announced the general availability of its Elasticsearch product on Google Cloud Platform (GCP), expanding its hosting options in four regions: US West (Oregon), US Central (Iowa), Europe (Belgium), and Europe (Frankfurt). This partnership combines Google's prominence in internet search with Elastic's popular open-source, distributed search engine, enabling users to deploy, manage, and scale Elasticsearch clusters on GCP efficiently. Elastic Cloud on GCP differentiates itself with unique features such as X-Pack, technical support from Elastic engineers, one-click upgrades, 30-minute snapshots, and custom plugin support. The subscription starts at $45 per month, with options to upgrade to premium plans, and Elastic plans to further expand into more GCP regions and enhance its offerings.

Elasticsearch 6.0.0-rc1, a release candidate version based on Lucene 7.0.0, has been launched as the fifth pre-release in the series leading up to the official 6.0.0 release, aimed at allowing users to test new features and provide feedback. It is intended solely for testing purposes, with no compatibility guaranteed for indices created in this version when upgrading to the general availability (GA) release. Significant improvements include the integration of the official Lucene 7.0.0, which brings enhancements such as space savings and better performance, as well as upgrades to X-Pack Security, including usability and documentation enhancements. Users are encouraged to test the release and report any issues, contributing to the ongoing development process.

The blog post by Jared Carey outlines the process of implementing Transport Layer Security (TLS) across the Elastic Stack, which includes Elasticsearch, Kibana, Beats, and Logstash, to ensure secure data transmission. The guide highlights the importance of TLS in protecting sensitive data from potential cybersecurity threats, such as compromised routers that can expose unencrypted data. It provides detailed instructions on setting up TLS, including generating certificates using the certgen tool provided by the X-Pack plugin, configuring Elasticsearch nodes, and setting up Kibana, Logstash, and Filebeat for encrypted communications. The post emphasizes the need for a certificate authority to sign certificates and explains the process of creating server certificates with specific DNS names to ensure proper identity verification. Additionally, it discusses the setup of roles and users within Kibana to secure the logging system further. While the instructions reference deprecated features, the article serves as a comprehensive guide for implementing TLS encryption in the Elastic Stack to enhance cybersecurity measures.

Elasticsearch 5.6.2, built on Lucene 6.6.1, has been released as the latest stable version and is now available for deployment on Elastic Cloud, the company's Elasticsearch-as-a-service platform. Users are encouraged to download and test this new version, providing feedback via Twitter or the forum, and reporting any issues on GitHub. The release includes notes on changes and updates specific to Elasticsearch 5.6 and X-Pack 5.6.2.

Machine learning can be effectively applied to nginx log data to extract operational insights and identify website issues using Elastic's X-Pack Machine Learning tools. The blog post describes how specific machine learning configurations, such as "Single Metric Job" and "Multiple Metric Job" wizards, can simplify the process of anomaly detection and analysis of web traffic. By analyzing data from nginx logs, users can detect significant anomalies in visitor rates, HTTP status code changes, and unusual client behaviors, such as those from bots or attackers. The post highlights a notable example on February 27th, where anomalies were detected in both overall event rates and unique visitor counts, which were attributed to a configuration change causing operational issues. These insights can be leveraged for early detection and alerting, providing operations teams with timely information to address system behavior changes. Future Elastic Stack releases will offer pre-packaged configurations to simplify the deployment and extend the usefulness of these tools for end users.

With the release of Logstash 6.0.0-beta1, a new centralized pipeline management feature was introduced to simplify the operation of Logstash instances. This feature allows users to manage Logstash deployments through a management UI in Kibana, enabling the creation, editing, and deletion of pipelines that are stored in an Elasticsearch cluster. This centralization facilitates easier orchestration and management of Logstash configurations, which can become complex in large-scale deployments. Additionally, the feature supports multi-tenancy and self-service, allowing data flows to be isolated using separate pipelines. Security is emphasized, with access control managed via a logstash_admin role and recommendations for best practices, such as using environment variables for credentials and enabling TLS for secure communication. While the current feature lays a strong foundation, future enhancements are planned, including security auditing, monitoring UI integration, and advanced config editor support. Feedback is encouraged through the Pioneer Program, which offers incentives for input on the feature's development.

Elastic uses the OpenAPI specification, previously known as Swagger, to document its Elastic Cloud Enterprise (ECE) REST API, which facilitates cluster management and automation tasks. The OpenAPI specification serves as a language-agnostic interface, enabling both humans and computers to understand the API's capabilities without requiring source code access. This specification aids in generating REST clients in various languages, which is beneficial for creating custom software layers around ECE. The blog post details how to retrieve the OpenAPI specification through an API or documentation, and explains how to use tools like Swagger Editor and Swagger Codegen to generate clients in languages such as Java and Python. The process involves obtaining Swagger Codegen, running it with specific commands to generate client code, and using the resulting client to perform tasks like retrieving and creating Elasticsearch clusters. Despite some known issues with Swagger Codegen in certain languages, the specification's flexibility allows for alternative community generators or even custom development.

Elasticsearch 6.0 introduces significant improvements aimed at reducing index storage size, notably through the deprecation of the "all" field and enhancements in handling sparsely populated fields. The removal of the "all" field, which previously concatenated all values for easier searchability but consumed considerable storage due to data duplication, accounts for nearly 40% of the storage savings. The remaining 60% is attributed to updates in Lucene 7.0, which accompanies Elasticsearch 6.0, allowing for more efficient storage of sparse fields by using an iterator-based approach to avoid storing empty values, thus reclaiming extra storage space. These changes, requiring minimal user intervention beyond an upgrade, result in a 41.5% reduction in storage space when tested with Metricbeat, a tool that frequently deals with sparsely populated fields.

The blog post highlights memorable presentations from the Elastic{ON} 2017 conference, showcasing major advancements and discussions in the Elasticsearch community. Key topics include the introduction of machine learning capabilities to the Elastic Stack, detailed explorations of new Elasticsearch features like cross-cluster search and multi-word synonyms, and insights into the capabilities of Timelion for time series analysis. Presentations also cover security strategies at Slack, Walmart's use of Elastic Stack for retail analytics, and updates on X-Pack developments, including management, monitoring, and new Graph UI features. The post encourages readers to submit their own talk proposals for the upcoming 2018 conference and highlights opportunities for recognition through the Cause Award.

Elastic's Infrastructure team has developed a method for integrating Kubernetes with Hashicorp Vault to manage secrets securely while reducing operational burdens. The integration, aimed at simplifying the migration of applications to Kubernetes, involves connecting Kubernetes to Vault and exposing secrets to running applications using tools like kubernetes-vault and vaultenv. This setup ensures that sensitive data is centralized and securely communicated without being hardcoded into source code repositories. The process involves using periodic tokens and sidecar containers for token renewal, allowing applications to access secrets via environment variables with minimal modifications. The approach supports a cloud-agnostic architecture, enabling the team to run Kubernetes in various environments while maintaining secure access to Vault APIs. Despite the introduction of Hashicorp's native Kubernetes integration, the techniques discussed, including using vaultenv for non-native services, remain relevant for managing secrets in Kubernetes deployments.

Elasticsearch 5.6.1, based on Lucene 6.6.1, has been released as the latest stable version available for deployment on Elastic Cloud, the platform's Elasticsearch-as-a-service offering. Users are encouraged to download and try out this version, and provide feedback on Twitter or the forum, with any issues reportable on GitHub. The release comes with accompanying release notes for both Elasticsearch and X-Pack 5.6.1, while highlighting the changes introduced in version 5.5.

The blog post "How to PSD2: Modern Banking API Architectures with the Elastic Stack" by Loek van Gool discusses the transformative impact of the Revised Payment Service Directive (PSD2) on the European banking industry. PSD2 mandates banks to open their APIs for secure access to customer account and transaction data, encouraging a shift towards more open banking ecosystems. This directive, while introducing potential risks, offers significant opportunities for banks to enhance their service offerings by integrating third-party APIs and leveraging advanced technologies like the Elastic Stack. The Elastic Stack is highlighted for its scalability, analytics capabilities, and suitability for high-traffic, API-driven environments, enabling banks to provide intelligent, real-time financial services. The post underscores the strategic decisions banks face, from basic compliance to becoming comprehensive financial service providers, and emphasizes the importance of API management, security, and user experience in this evolving landscape.

In the "Brewing in Beats" blog post by Monica Sarbu, the latest updates in the Beats project, a part of the Elastic Stack, are highlighted. The post discusses the new capabilities of Auditbeat to run alongside the auditd daemon on Linux kernels 3.16 and above, thanks to support for multicast in go-libaudit and Auditbeat. The default configurations for the number of shards have been optimized, reducing the number of shards for typical installations, and changes have been made to ensure pod annotations in Docker and Kubernetes are retained longer for log reading purposes. The update also covers various fixes and enhancements across different Beats components, including Metricbeat, Packetbeat, Filebeat, and documentation improvements, all aimed at enhancing performance, usability, and configuration flexibility in the upcoming 6.0.0-rc1 release.

Elasticsearch's release of the Java High-Level REST Client aimed to address the limitations of the previously used native Java Client, which relied on a binary protocol that posed security and compatibility challenges. This new client, introduced with Elastic Stack version 6.0.0-beta1, builds upon the low-level REST client to offer higher-level request and response objects similar to those used by the native client, facilitating a smoother migration for users. Although still dependent on specific Elasticsearch versions, the high-level client supports essential APIs like index, bulk, get, delete, and search, with plans to expand its capabilities based on user feedback. The Elasticsearch community is encouraged to trial this client, report issues, and participate in its ongoing development, with the ultimate goal of replacing the transport client entirely.

Creating threshold alerts in Elasticsearch has been simplified, catering to requests for a user-friendly interface at the Elastic{ON} '17 conference. Although designing a universal alert UI is challenging due to the varied nature of alerts, the focus was placed on developing a straightforward UI for threshold alerts, the most requested type. This feature requires Elasticsearch and Kibana version 6.0.0-beta 2 or higher, with X-Pack installed, and involves using Metricbeat to gather system data for alert creation. Users can configure alerts in Kibana's Watcher, specifying conditions like CPU usage exceeding 50% in the last five minutes, and set actions such as sending an email notification. Future plans include integrating Kibana's saved searches as inputs for alerts and expanding UI options for different alert scenarios. Feedback from users is encouraged through the Pioneer Program to enhance software development.

In a 2017 blog post, authors Thomas Neirynck and Peter Pišljar introduce two visualization types in Kibana version 5.5: Region Maps and Gauge charts. Region Maps, or Choropleth maps, use color gradients to represent metric values and can be configured by linking terms aggregation results to GeoJSON FeatureCollections, with warnings available for unmatched items. Users can also add custom GeoJSON layers by configuring the kibana.yml file, though CORS settings may need adjustments. Gauge and Goal charts, while similar in appearance, have different purposes; Goal charts track progress toward a fixed goal, while Gauge charts monitor current values within specified ranges. The authors also note upcoming changes in Kibana's major v6 release, which include increased flexibility in rendering technologies, allowing the use of React or JavaScript instead of Angular, and the separation of visualization configuration from the visualizations themselves, which will require minor plugin edits.

The Elastic Stack ArcSight Integration introduces a streamlined process for integrating ArcSight Security Information and Event Management (SIEM) data with the Elastic Stack, facilitated by the new Logstash ArcSight module. This integration allows users to ingest, analyze, and visualize ArcSight's Common Event Format (CEF) data using the Elastic Stack's powerful tools, such as Kibana, for enhanced security analytics. The Logstash ArcSight module simplifies the process, enabling users to explore their security events in minutes with just one command. This integration addresses the growing demand for robust security analytics by enabling real-time threat detection and analysis across vast volumes of data. It also leverages Elastic's machine learning capabilities to enhance automated detection of attack behaviors. The module is free to use with a basic Elastic license, with options to upgrade for additional features, and represents a significant technology partnership between Elastic and ArcSight, though each company remains independent in terms of product sales and distribution.

Elasticsearch 5.5.3, built on Lucene 6.6.0, has been released as the latest stable version and is available for deployment on the Elastic Cloud platform. Users are encouraged to download this version, explore its features, and provide feedback via Twitter or the forum. Additionally, any issues encountered can be reported on the GitHub issues page. The release notes and breaking changes for Elasticsearch 5.5 and X-Pack 5.5.3 are also available for review to assist users in understanding updates and modifications.

Elasticsearch 5.6.0, based on Lucene 6.6.0, introduces several significant updates, including the ability for rolling upgrades to version 6.x, enhanced deprecation logging, and the integration of migration assistance features into X-Pack, available at no cost. This release also unveils a new high-level Java REST client to simplify interactions for Java developers, which will eventually replace the Transport Client in version 7.0. The update further addresses parent/child relationships with a new join datatype and improves search scalability by implementing a shard prefiltering phase to optimize search efficiency and prevent cluster overload. These enhancements aim to streamline the upgrading process and improve system performance, ensuring a seamless transition to future versions.

Elastic Cloud, the official hosted service for Elasticsearch and Kibana, provides users with a platform to manage and monitor system metrics via tools like Metricbeat. Users can start with a 14-day free trial requiring only an email for sign-up. The process begins by creating an Elasticsearch deployment, customizing it with various templates and configurations, and generating necessary access credentials. Users can enhance security by setting up a new superuser account through Role-Based Access Control. Metricbeat, a tool for collecting and shipping system metrics, is configured to send data to the Elastic Cloud by modifying the metricbeat.yml file with the appropriate Cloud ID and authentication details. Once set up, Metricbeat facilitates the visualization of metrics through pre-loaded Kibana dashboards, enabling users to efficiently monitor system performance and deploy the tool across multiple systems.

Elastic{ON} 2018 is set to gather the global Elastic community in San Francisco, with a call for presentations inviting speakers to share their experiences using the Elastic Stack and other distributed systems. The event, known for featuring stories from both large and small companies, is seeking new speakers to contribute by October 31, 2017. This year, several improvements have been introduced to support applicants, including the Elastic{ON} SpeakerOps StandUp Series, which offers guidance and inspiration from past speakers and the CFP decision panel, as well as examples of top-rated talks. The event encourages submissions on various topics, even those not directly related to the Elastic Stack, and offers financial aid to cover travel costs, ensuring that financial constraints do not hinder participation.

Elastic has simplified the process for new users to experience the Elastic Stack by deploying a fully-featured stack using Docker Compose with a single command. This initiative aims to shorten the learning curve from the initial discovery of Elasticsearch to exploring its capabilities, like Kibana drill-downs, without the complexities of production setups. The approach involves using Docker Compose to orchestrate various containers, including Elasticsearch, Kibana, and multiple Beats modules, alongside external services like NGINX, Apache2, and MySQL, to allow comprehensive monitoring and data ingestion. The setup is designed for exploratory purposes and is not recommended for production. It automatically populates dashboards with data from system logs and network traffic, providing a rich interactive experience for users. Elastic encourages contributions and customization, allowing users to adapt the environment to their needs and expand its functionality with additional modules.