March 2020 Summaries

In response to the COVID-19 pandemic, the Elastic community, which includes a diverse range of sectors such as hospitals, media, and online retailers, is encouraged to share their experiences and adaptations using the Elastic Stack through the #ElasticStories hashtag. This initiative aims to connect businesses and organizations within the community, allowing them to exchange best practices and innovative solutions during these challenging times. Originally launched to share Elastic Stack stories, the hashtag's focus has temporarily shifted to highlight how various entities are utilizing the platform to address the demands of the pandemic, maintaining a commitment to fostering a free and open community.

Version 7.6.2 of the Elastic Stack has been released, bringing a series of fixes and minor enhancements to the stack's components. Among the updates, a significant security fix addresses a privilege escalation flaw that previously allowed an attacker to generate an API key with elevated privileges. Users are encouraged to upgrade to this latest version to benefit from these improvements. Detailed information on all changes for each product within the stack can be found in the release notes for Elasticsearch, Kibana, Logstash, and Beats.

Version 6.8.8 of the Elastic Stack has been released, introducing important fixes and minor enhancements to the platform. Among the key updates is a significant security patch addressing a privilege escalation flaw where an attacker could generate an API key with elevated privileges. Users are encouraged to upgrade to this latest version to benefit from these improvements. For detailed information on changes across individual products like Elasticsearch, Logstash, Kibana, and Beats, users are advised to consult the release notes.

Instrumenting a polyglot microservices application with Elastic APM (Application Performance Monitoring) can significantly enhance observability and performance insights. Using an example application, the process involves setting up Elastic Stack with APM to monitor various services written in different languages like VueJS, Go, NodeJS, Java, and Python. Each service is equipped with Elastic APM agents tailored to its language, enabling distributed tracing and logging for comprehensive system monitoring. Elastic APM tracks transactions, spans, and logs across services, allowing for quick identification and resolution of performance bottlenecks. The setup utilizes Docker containers for deployment and demonstrates diverse ways to configure agents and instrument applications, highlighting the flexibility and depth of Elastic APM. This approach not only improves user experience and business efficiency but also fosters collaboration among development, operations, and security teams by providing a unified perspective on application performance.

Elastic SIEM, now part of the broader Elastic Security solution, offers a free and open-source application designed to enhance the capabilities of security teams by providing visibility, threat hunting, automated detection, and SOC workflows. Built on the Elastic (ELK) Stack, it leverages the speed and scalability of Elasticsearch, integrating tools like Elastic Maps and Kibana to facilitate situational awareness and threat detection. The application includes detection rules aligned with the MITRE ATT&CK framework and offers unique features such as a timeline investigator and investigation templates to reduce mean time to detect and respond to threats. Elastic SIEM's open approach extends beyond code, fostering a community-driven environment with an open roadmap and data model, enabling organizations to deploy and scale their security operations without the financial constraints of traditional SIEM licensing models. Elastic also provides commercial extensions to further enhance capabilities, including machine learning-based anomaly detection and integration with external systems, all while maintaining transparency and community engagement through forums and public issue tracking.

Elastic APM has embraced the W3C TraceContext standard for context propagation in distributed tracing, which is crucial for monitoring microservice architectures by allowing different services to communicate and report their traces in a unified manner. Context propagation ensures that each service can uniquely identify and contribute to a trace, and adopting a unified format allows multiple APM vendors to monitor the same architecture without loss of observability. The W3C TraceContext specification, now a W3C recommendation, defines the traceparent and tracestate HTTP headers to facilitate this process, enabling services to report trace components and visualize them in a comprehensive trace view. Elastic APM was an early adopter of this standard and has updated its implementations to align with the official specification, allowing users to identify performance bottlenecks and errors by easily integrating with their applications.

Elastic announced its participation in the Google Summer of Code (GSoC) 2020, inviting university students to collaborate on open-source projects, specifically focusing on the Elastic UI Framework (EUI). EUI, originally developed as the design system for Kibana, has evolved to be widely used both within and outside Elastic, benefiting from global contributions and rapid feature releases. Elastic emphasizes the importance of maintaining robust support for diverse projects and platforms while providing comprehensive documentation to aid users. Students participating in GSoC will be mentored throughout their three-month coding journey, defining project deliverables and receiving continuous feedback, with successful contributions potentially being merged into the broader EUI project. The deadline for project proposals is March 31, 2020, and interested candidates are encouraged to engage with Elastic's resources and support channels to refine their submissions.

BlackSky leverages a diverse array of data sources, including satellite imagery, social media feeds, and IoT sensors, to provide near real-time business analytics through an Elastic-powered analytics engine. By integrating machine learning, artificial intelligence, and computer vision, BlackSky's system efficiently processes and cleans data, identifying trends, patterns, and anomalies to deliver timely intelligence. CTO Scott Herman demonstrated at an Elastic{ON} Tour event in Washington, DC, how this technology can be used to forecast changes in commodity prices, detect construction delays, and anticipate political unrest. This integration of multiple data streams allows BlackSky to offer its customers actionable insights with impressive speed and accuracy.

Elastic Workplace Search aims to centralize information and content from various organizational platforms into a single source of truth, enhancing work efficiency by providing a unified search experience. It connects easily to popular content platforms like Dropbox, Salesforce, and Zendesk, allowing seamless synchronization of data across collaboration tools. For unique organizational needs, custom sources can be created, functioning as individual repositories with their own data structures, supporting real-time updates, content filtering, and keyword detection without additional configurations. Custom sources are integrated into the search experience like standard data sources, enabling tailored relevance and access control for different teams through features such as Source Prioritization. Additionally, users can update data structures and result layouts through an intuitive interface, ensuring the search experience remains flexible and maintainable over time.

The Elasticsearch Service has expanded its availability to the Google Cloud Platform (GCP) region in São Paulo, marking its first presence in South America and its 11th GCP region globally. This move allows existing users to access the new region immediately, while new users can explore the service with a 14-day free trial. The service offers advanced features such as machine learning, index lifecycle management, and intuitive visualization tools like Kibana Lens and Canvas. Elastic and Google have partnered to enhance the service with native GCP console integration and offer it through the GCP Marketplace with integrated billing options. Additionally, Elastic provides resources and support for users migrating their Elasticsearch data to the new region.

Elasticsearch Service has expanded its availability to the Google Cloud Platform (GCP) region in Singapore, marking it as the 12th GCP region globally and the 4th in the Asia Pacific area. This allows existing users to immediately access the Elasticsearch Service on Elastic Cloud within the new region, while new users can explore the service through a free 14-day trial. The service offers advanced search, observability, and security solutions, alongside features like machine learning, index lifecycle management, and intuitive visualization tools such as Kibana Lens and Canvas. Elastic's partnership with Google aims to enhance native GCP console integration and expand the service to more regions, and customers can subscribe through the GCP Marketplace, benefiting from integrated billing. The company has also provided resources for users looking to migrate existing Elasticsearch data, offering documentation and webinars to assist in the transition.

Elasticsearch Service on Elastic Cloud has introduced a beta availability for AWS GovCloud (US East), aimed at users handling government data and seeking FedRAMP Moderate Impact level authorization, which ensures robust security compliance. The service streamlines deployment by providing a step-by-step guide to create and manage Elasticsearch clusters, access Kibana, and incorporate extensive search, observability, and security capabilities using customizable deployment templates. Users can configure data management strategies like hot-warm templates and utilize the SIEM app for enhanced visibility and automated detection. The service facilitates enterprise search through Elastic App Search, offering developer tools and real-time analytics, while observability features allow centralization and management of logs, metrics, and APM data. Security measures include leveraging lifecycle management for data archiving. The beta period supports production workloads with options for annual subscriptions, promising future availability of monthly plans.

The Elasticsearch and Apache Lucene update for March 13, 2020, highlights several advancements, including the introduction of Async search APIs and Data Streams. The Async search APIs, currently in incubation, aim to enhance Kibana by allowing asynchronous search requests, enabling users to bypass the 30-second timeout starting with release 7.7 and improving dashboard functionality in subsequent versions. Data Streams formalize time series data management, offering a new configuration and API to improve user interaction and internal management without relying on aliases. Additionally, Index Templates v2 introduces "component templates" to resolve issues with merging existing templates, while recent performance testing revealed regressions linked to changes in Apache Lucene and Elasticsearch's query cancellation. Furthermore, enhancements in geometry queries were made, optimizing CPU usage by specializing shape calculations, and SimpleFSDirectory is deprecated due to redundancy with NIOFSDirectory and performance considerations across operating systems.

In response to the COVID-19 pandemic, Elastic has implemented several measures to ensure the safety and well-being of its employees while maintaining operational continuity. All in-person events, including leadership offsites and ElasticOn Tours, have been transitioned to virtual formats, with plans to further virtualize community interactions and workshops. The company has mandated global remote work from March 13 to March 27, emphasizing its distributed work model and offering a home working allowance to facilitate employees' transition. Additionally, travel has been restricted in line with local guidelines, and a specific time-off category has been introduced for non-illness-related absences due to COVID-19, such as school closures. Public in-facility training sessions have shifted to virtual formats, and employees are encouraged to stay informed through a dedicated email and Slack channel.

Elastic has introduced a new resource-based pricing model for its Enterprise Search products on Elastic Cloud, aiming to address the complexities and hidden costs associated with traditional search solution pricing. Unlike previous models that often rely on arbitrary criteria like the number of documents, queries, or users, Elastic's new approach charges customers based on actual resource consumption, such as RAM, disk, and CPU. This model allows for more transparent and scalable pricing, enabling customers to pay only for the resources they use and easily scale their search capabilities as their needs grow. By eliminating the need for complex contracts and pricing tiers, Elastic's approach is designed to be more fair and cost-effective, potentially resulting in lower costs for customers over time. This shift in pricing strategy reflects Elastic's commitment to customer-first principles, offering a straightforward and flexible solution that aligns costs with actual usage and encourages experimentation with their services.

The Elastic Security platform, formerly known as Elastic SIEM, has introduced a modern detection engine as part of Elastic Security 7.6, offering a unified SIEM rule experience for security operations centers (SOCs). This detection engine leverages Elasticsearch analytics and runs on Kibana's distributed execution platform, enabling the creation and management of signals, which are documents generated when rule conditions are met. These signals help practitioners manage security alerts efficiently, allowing them to analyze and close signals after investigation. The platform supports a streamlined workflow for rule creation, offering prebuilt rules and customizable settings, including integration with MITRE ATT&CK tactics. The detection engine is scalable, utilizing Kibana's Alerting framework and task manager to balance tasks across instances, and includes mechanisms to prevent duplicate signals. Users can engage with the Elastic community to provide feedback and influence future developments, such as incorporating machine learning and advanced queries.

The Kibana team, led by Raya Fratkina, is actively working on several enhancements and updates to the platform. Key developments include the transition of Discover’s result tables to the EuiDataGrid component, promising a more user-friendly layout akin to the Lens interface, which addresses user requests and sets the stage for future improvements. Additionally, the team has integrated query cancellation and asynchronous search capabilities into Kibana, introducing a new search strategy that facilitates asynchronous requests and enhances efficiency by managing Elasticsearch connections. The platform also enhanced its search functionality to support frozen indices and rollups while removing reliance on EsClient. For Kibana users, upgrades are now more robust, even with ongoing Elasticsearch snapshots, and there are UI updates to assist administrators with deprecated roles. Plugin developers benefit from the ability to register SavedObject types, and changes in UI application integration require minor adjustments. Moreover, new APIs for internal metrics and optional authentication for HTTP routes have been introduced, alongside enhancements for plugin developers to improve routing and security features. The team is also hiring for several roles, including Senior JavaScript Engineer and Principal Product Manager positions.

Elastic has expanded its Maps Service by adding new data layers that complete the European continent for second-level national boundaries, including countries like Albania, Greece, and Ukraine, bringing the total number of datasets to 65. These layers are accessible for Kibana region map visualizations, the Maps app, and are available for download on the Elastic Maps Service page, complete with alphanumeric data like ISO codes and names in multiple languages. The data, curated from OpenStreetMap and Wikidata, is designed for integration with business data using ISO codes as join identifiers, and can be accessed via the Elastic Maps Service or downloaded for air-gapped environments. For those interested in direct data usage, GeoJSON upload and tools like GDAL can facilitate data integration into Elasticsearch, offering a resource for data enrichment and regional aggregation. Elastic encourages users to explore these new features with a free trial of the Elasticsearch Service.

Elastic CEO Shay Banon provided an internal update on the company's response to the COVID-19 pandemic, outlining a series of measures to adapt to the changing circumstances. All physical presence for upcoming events such as the SLT+ELT Offsite, Field Leadership Offsite, and ElasticON Tour have been canceled and transitioned to virtual formats, with adjustments made to accommodate various time zones. Internal travel has been suspended, and any travel across countries or regions requires managerial approval, while employees are encouraged to work from home, with office access determined by local guidelines. Elastic is implementing a new time-off category in Workday for COVID-19 related non-illness absences, ensuring that it does not affect vacation or annual leave. Public in-facility training sessions are also being moved online for the foreseeable future. Banon emphasized the company's existing strengths in being a distributed organization and expressed gratitude to employees, customers, and partners for their resilience and adaptability during the crisis.

Flávio Knob, a financial auditor and software developer for the state government in Brazil, was named the first Elastic Certified Professional of the Year, an accolade recognizing his exemplary expertise and community contributions related to the Elastic Stack. This prestigious award highlights his role in mentoring others, his outstanding application, and numerous recommendations from peers. Flávio discovered the Elastic Stack while enhancing a website's search functionality for his employer and has since become a significant figure within the Elastic community, organizing meetups, writing articles, and contributing to Elastic projects. His efforts have not only improved his workplace’s operations but also solidified his status as a key resource and leader in the field. The award was presented at the Elastic{ON} Tour event in San Francisco, marking a notable achievement in Flávio's career and further opening up new possibilities for him.

In this seventh installment of the Elastic SIEM for home and small business blog series, the focus shifts to reviewing data collected through Elastic SIEM version 7.4, despite the general availability of version 7.6. The series illustrates how to set up and utilize Elastic SIEM for monitoring and securing network activities for personal or small business use by collecting data from various operating systems and network devices. The blog guides users through configuring and using the SIEM application to analyze data from different hosts and networks, highlight authentication failures, and create timelines for security events. It emphasizes the importance of upgrading to newer versions like 7.6 for improved features and bug fixes, such as the resolved GeoIP issue. The series encourages readers to explore further data collection options using Logstash Input Plugins and Filebeat Modules and to utilize Elastic's documentation and support for overcoming any challenges encountered.

The introduction of meta engines in Elastic App Search with the 7.6 release offers a significant advancement for companies looking to unify and scale their search capabilities across multiple engines. Meta engines allow for the creation of a comprehensive search experience by aggregating results from individual sub-engines, each maintaining its own rules and autonomy. This innovative feature enables organizations to streamline user experiences by reducing the fragmentation caused by separate engines for different departments like customer support, marketing, and documentation. Despite the added complexity in managing search functionalities, meta engines provide enhanced flexibility and control for search administrators, who can establish custom rules and relevance settings independently of the sub-engines. Role-based access controls further support secure management of user permissions across engines. Available for both Elastic Cloud and self-managed versions, meta engines are designed to be cost-effective under Elastic’s new pricing model, which charges based on resources consumed rather than the number of users or operations.