September 2021 Summaries

Elastic has launched a free, one-year virtual training program for U.S. veterans and active duty military personnel as part of their Operation Giving Back initiative. Aimed at helping military members transition to civilian careers, the program offers on-demand training in Elastic technologies, which are in demand across various industries. Participants can access resources such as Quick Start guides, training fundamentals, and courses like Data Analysis for Kibana, with support provided throughout their learning journey. Elastic's commitment to the military community, bolstered by its existing connections with the Department of Defense and organizations like VetsinTech, highlights the company's dedication to empowering veterans with skills for the global job market.

The announcement highlights the integration of direct data ingestion from Google Cloud Storage (GCS) to the Elastic Stack using Google Dataflow, simplifying the data pipeline process for developers, site reliability engineers, and security analysts. This collaboration between Google and Elastic allows users to easily transfer logs and events stored in GCS to the Elastic Stack with minimal effort, reducing operational overhead and speeding up troubleshooting. The integration supports CSV file formats and plans to include JSON support, streamlining data ingestion without the need for custom data processors. This enhancement is applicable to all Elastic Stack users, whether on Elastic Cloud, Elastic Cloud in the Google Cloud Marketplace, or self-managed environments. An example provided involves analyzing earthquake data from the USGS, demonstrating the simplicity of data ingestion and visualization using Google Cloud Console and Kibana. This initiative aligns with Elastic's goal of offering flexible solutions and enhancing user experience across platforms.

The recent integration between Google BigQuery and the Elastic Stack allows for seamless data ingestion using Google Dataflow, simplifying the process for data analysts and developers. This collaboration between Google and Elastic enables users to transfer data from BigQuery to Elastic without needing additional data shippers or ETL tools, reducing operational overhead. The integration leverages Dataflow templates, allowing users to easily ingest data with a few clicks in the Google Cloud Console, eliminating the need for separate data processors like Logstash. This setup facilitates various use cases, such as full-text search and dashboard visualization through Elastic's Kibana. Users can start ingesting data by selecting the BigQuery to Elasticsearch template, filling in required parameters, and running a Dataflow batch job, thus enhancing the capability to analyze data effectively within Elastic's environment. This development underscores Elastic's commitment to providing frictionless solutions, showcasing the value of Elastic Cloud and the enhanced experience it offers through streamlined integrations with platforms like Google Cloud.

Elastic's Infosec Detections and Analytics team, known as Customer Zero, outlines procedures for configuring the Elastic Security app and Machine Learning jobs to utilize Cross Cluster Search (CCS) effectively. The process involves adjusting the Security app's index patterns to incorporate CCS and modifying the built-in detection rules to use CCS patterns via duplication and bulk editing. For Machine Learning, datafeeds must be updated to adopt CCS patterns, a task requiring API access since it cannot be done directly through the UI. The post emphasizes ensuring all clusters are updated to version 7.14 or newer to support Elastic Query Language with CCS and provides guidance on managing and tracking rules through version updates. The team concludes by hinting at future updates on their ongoing use of Elastic products for security.

The article highlights the challenges and opportunities for Chief Information Officers (CIOs) in leveraging cloud technology to harness the vast amounts of data that businesses store but seldom use effectively. It emphasizes that despite a rapid growth in cloud data storage, most companies fail to utilize over two-thirds of their data, which hinders the creation of data-driven organizations and transformational business outcomes. To address this, CIOs are urged to focus on data integration, observability, search, and security within cloud infrastructure to unlock value and drive growth. By consolidating data from disparate sources and adopting observability tools that offer real-time insights, organizations can improve decision-making and enhance customer experiences. Additionally, the implementation of advanced search techniques and a Zero Trust security framework can further enable companies to navigate cyber risks and safeguard their data. The article underscores the need for strategic initiatives that modernize data management and protection, ultimately allowing businesses to become truly data-driven.

In light of increasing stress levels and cybersecurity threats, CISOs in 2022 are encouraged to focus on five key priorities to maintain security and balance: adapting to complex hybrid work environments, accelerating the adoption of Zero Trust frameworks, automating security workflows, upskilling security teams, and preventing burnout. The transition to hybrid work presents unique security challenges, necessitating investments in cloud-based security solutions like extended detection and response (XDR). The Zero Trust framework, highlighted by President Biden's executive order, emphasizes a "trust nothing, record everything" approach, requiring security teams to scrutinize network access and products more closely. Automation is critical in managing complex security systems, freeing up human analysts for high-level threat analysis. Upskilling involves not only technical expertise but also business acumen and communication skills, as security professionals need to integrate seamlessly with management teams. Amidst these challenges, CISOs must ensure work-life balance for their teams and themselves to prevent burnout and maintain effectiveness in addressing the demands of a hybrid workforce.

Hybrid work models, which gained prominence during the pandemic, are proving instrumental in assisting digital transformation projects that often falter due to cultural challenges. Despite the high failure rate of such projects, many executives persist with traditional strategies focused on technology rather than cultural adaptation. Michael Watkins, a leadership consultant, highlights the necessity of equipping executives with the skills to rapidly change organizational culture to support digital initiatives, emphasizing collaboration and communication. The shift to remote work has led to increased collaboration and transparency, with 70% of IT leaders noting improved synergy between business and technology teams, according to KPMG. Companies that have embraced hybrid work models are finding themselves better positioned to succeed in digital transformation by fostering a collaborative culture, as seen in examples like Elastic, which promotes transparency and decentralized decision-making. Cultural transformation requires dedicated resources, executive sponsorship, and clear communication to ensure that employees understand the importance of their roles in the changes being implemented, ultimately leading to improved employee connection, loyalty, and organizational success.

Elastic and Google Cloud have enhanced their collaboration, enabling developers, site reliability engineers, and security analysts to seamlessly ingest data from Google Pub/Sub into the Elastic Stack via Google Dataflow templates. This integration simplifies the data pipeline by allowing direct data streaming without the need for installing data shippers like Filebeat, Elastic Agent, or Fluentd, thus reducing operational overhead. The process is streamlined through a few clicks in the Google Cloud Console, where users can configure Dataflow jobs to send logs and events from Google services such as Google Cloud Audit, VPC Flow, or firewall logs to the Elastic Stack. This agentless method leverages Google Dataflow's serverless capabilities to replace traditional log processing methods, offering users a more efficient way to gather insights from their data. The integration works across different Elastic environments, whether on Elastic Cloud or self-managed, and is designed to enhance the overall experience for joint users of Google Cloud and Elastic services. Moreover, Elastic provides prebuilt assets in Kibana to help users maximize the value of the ingested data, making the Elastic Stack more accessible and versatile for various operational needs.

The French Ministry of Agriculture and Food has successfully implemented the Elastic Stack to enhance the monitoring and management of the commercial fishing industry, aiming for better data traceability, quality, and analysis. By using Elastic's capabilities, the Ministry can ingest vast amounts of data, such as declared fish hauls and GPS tracking, to provide verified and real-time information that assists in enforcement actions and negotiations over fishing rights. Elastic was chosen over competitors like Splunk and Graylog for its speed, ease of use, and scalability, with tools like Kibana and Canvas offering advanced data visualization options. This system supports compliance with protective fishing regulations by tracking vessel activities and identifying law infringements, with data stored for up to ten years encompassing over 135 million records. The Ministry plans to expand Elastic's application beyond fishing to include broader food tracking initiatives, signifying its potential for addressing complex political, economic, and environmental challenges.

The Elastic APM iOS agent has been released as a technical preview by Elastic to gather community feedback and provide initial functionality within the Elastic Observability stack, though it is not yet intended for production use. This agent is built on the opentelementry-swift SDK, allowing it to capture frameworks and libraries instrumented with Open Telemetry, and it provides auto-instrumentation for aspects like URLSession, CPU and memory usage, and network connectivity. Offered through the Swift Package Manager, the agent requires Swift v5.3 and iOS v11, supporting on-prem or cloud solutions with a slim API for configuration. Elastic hopes to engage the community for future development and improvements by encouraging participation through their discussion forum and GitHub repository.

Elastic Security 7.15 introduces extended detection and response (XDR) capabilities to enhance protection against various cyber threats by implementing malicious behavior analytics, memory threat protection, and host isolation across multiple operating systems, including Linux, Windows, and macOS. The update strengthens defenses against advanced attack techniques, such as memory manipulation and credential theft, by mapping protections to the MITRE ATT&CK framework and ensuring the ability to quarantine compromised systems. Additionally, new integrations with platforms like Carbon Black EDR, CrowdStrike Falcon, Cloudflare, Hashicorp Vault, and Palo Alto Networks Cortex XDR enhance data ingestion and analysis, providing security teams with comprehensive visibility across their attack surface. The update also improves alert triage processes with enhanced filtering and interaction options, while osquery enhancements offer standardized query results and more granular access controls, enabling organizations to better manage and respond to security incidents.

Elastic Enterprise Search 7.15 brings significant enhancements, including the general availability of App Search's web crawler, which simplifies the setup of powerful search experiences by streamlining data ingestion and content extraction. This release introduces features such as automatic crawling controls, content extraction utilities, and support for robots.txt and sitemaps, ensuring efficient and reliable web crawling. Additionally, Workplace Search now offers increased personalization with custom branding options, enhanced sync configurability, and natural language query capabilities, allowing organizations to tailor search experiences to their specific needs. The platform supports various data integrations, including Google Drive and Slack, and provides tools for creating bespoke search integrations within high-traffic applications. Elastic Enterprise Search 7.15 is available on Elastic Cloud and as a self-managed version, offering flexibility for users to explore these new capabilities.

ElasticON Global 2021 is a free, virtual conference hosted by Elastic from October 5-7, focused on celebrating and empowering problem-solvers tackling complex global challenges. The event, themed "Solve," features a diverse lineup of keynotes, fireside chats, and over 100 technical sessions with insights from industry leaders like Adobe, Microsoft, and NASA's Jet Propulsion Laboratory. Highlights include discussions on digital transformation, cybersecurity, and cloud innovation, with notable speakers such as Elastic founder Shay Banon and chef José Andrés. Attendees can participate in skill-building workshops, live training courses, and engage with a virtual exhibit hall featuring sponsors like Microsoft Azure and Google Cloud. The conference also includes the Elastic Excellence Awards to honor impactful projects and offers donation opportunities to World Central Kitchen and Girlstart.

Elastic 7.15 introduces significant enhancements to the Elastic Search Platform, including Elasticsearch and Kibana, and its integrated solutions: Elastic Enterprise Search, Elastic Observability, and Elastic Security. This release features the general availability of the Elastic App Search web crawler and improved Google Cloud integrations, facilitating faster data ingestion and the creation of robust web search experiences. Elastic Observability's new APM correlations feature aids DevOps teams in accelerating root cause analysis by identifying attributes linked to high-latency transactions. Elastic Security enhances its extended detection and response capabilities with malicious behavior protection across multiple operating systems and introduces one-click host isolation for cloud-native Linux environments, leveraging eBPF technology. Additionally, Elastic Cloud now supports Google Cloud Dataflow and Private Service Connect, streamlining data ingestion and ensuring secure connectivity within the Google Cloud network. These advancements aim to improve search, observability, and security processes, offering users a more efficient and secure experience on the Elastic platform.

Elastic Observability 7.15 introduces key enhancements such as automated correlations for efficient root cause analysis, frictionless log ingestion from Google Cloud Platform (GCP), and unified observability across application services. The update includes the general availability of automated correlations that help surface attributes correlated with high-latency or erroneous transactions, thereby streamlining troubleshooting processes for DevOps and SRE teams. The new version also provides agentless log ingestion from GCP, enhancing monitoring capabilities for native Google Cloud services, and integrates new data sources, including JRuby and Azure Spring Cloud logs. Additionally, Elastic Observability 7.15 offers improved visualization and workflow capabilities, with enhanced transaction latency distribution and trace selection features, along with the ability to visualize external dependencies, such as backends and caches. The self-managed version of the Elastic Package Registry is now available for air-gapped deployments, providing flexibility for environments with network restrictions. These developments underscore Elastic's commitment to providing comprehensive observability solutions that enhance the performance and reliability of modern applications.

The 7.15 release of Elastic Cloud introduces several enhancements to the Elastic Stack, with a focus on improving data ingestion, analysis, and performance. Key updates include new integrations with Google Cloud services, which facilitate direct data ingestion into Elastic Cloud and enhance network security through Google Cloud Private Service Connect. The release also features improvements in Elasticsearch, such as reduced data transfer traffic and new APIs for optimizing performance, as well as the introduction of composite runtime fields in Elasticsearch and Kibana for more efficient data management. Kibana 7.15 enhances user interaction with a preview pane for runtime fields and improved chart customization, while machine learning capabilities are expanded with easier management and monitoring of jobs. Additionally, Elastic Cloud now supports cost-effective ARM-based Graviton2 instances on AWS, offering better price performance. These updates collectively aim to streamline data operations, enhance security, and improve the usability of Elastic's services across various platforms.

Version 6.8.19 of the Elastic Stack has been released, offering fixes and minor enhancements across its products, including Elasticsearch, Kibana, Beats, and Logstash. Users are encouraged to upgrade to this latest version to benefit from these improvements. Detailed information on the changes for each product can be found in the 6.8.19 release notes.

Version 7.14.2 of the Elastic Stack has been released, featuring various fixes and minor enhancements across its components. Users are encouraged to upgrade to this latest version to benefit from these improvements. The update impacts several products, including Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Elastic Observability, APM, and Elastic Security. For a comprehensive overview of all the changes and updates included in this release, users are directed to consult the detailed release notes.

Elastic Stack has released its second public alpha version, 8.0.0-alpha2, following the first release a month prior, with significant updates including 580 pull requests for Elasticsearch alone and a planned upgrade to Lucene 9. Users are cautioned that this alpha version is not suitable for production environments and may not be compatible with future releases or the eventual general availability version. The release is not yet available on Elastic Cloud, but a preview version is expected soon. Elastic encourages users to participate in its Pioneer Program by testing this alpha version across various components such as Elasticsearch, Kibana, Beats, Logstash, and APM, and reporting any bugs to help enhance the release's robustness. Participants in the program may receive unique rewards once a stable version is released.

Elastic Cloud on Kubernetes (ECK) 1.7 simplifies the monitoring of the Elastic Stack by allowing users to specify a reference to a monitoring cluster, which automatically sets up sidecar containers to ship logs and metrics to the designated Elasticsearch cluster. This update reduces the complexity associated with Beats configuration and Kubernetes role-based access control. Users can deploy ECK in Kubernetes, set up a monitoring cluster first, and then link a production cluster to it by referencing the monitoring Elasticsearch cluster in the specification. ECK also provides out-of-the-box Kibana alerts based on best practices, although users can customize these alerts to suit their needs. However, it's important to note that the monitoring cluster must be separate and managed by ECK within the same Kubernetes cluster as the monitored cluster, as an Elasticsearch cluster cannot monitor itself.

Elastic celebrates Elasticians Unidos Month, an extension of the US Hispanic Heritage Month, to honor and appreciate Hispanic and Latinx communities globally. This initiative involves virtual events such as cooking, folklore, music, and cultural storytelling, allowing Elastic employees to share and celebrate their diverse heritages. Employees like Ully Sampaio, Johana Ochoa, Javier Detrinidad, Kimberly Chavez, and Marcela Rysbekov share their experiences of bringing their cultural backgrounds to work, emphasizing the inclusive and supportive environment at Elastic. They highlight how their personal histories enrich their professional roles, enhancing cultural understanding and business practices within the company. The celebration underscores Elastic's commitment to diversity, equity, and inclusion, creating a workplace where differences are embraced and valued, thus fostering a sense of belonging and community among employees.

The 7.14 release of Elastic Maps introduces enhanced geospatial capabilities, allowing sysadmins, cartographers, and dashboard designers to tailor maps for richer geodata storytelling. Available on Elastic Cloud and for self-managed setups via Elastic Cloud Enterprise and Elastic Cloud for Kubernetes, Elastic Maps now features simpler configuration and scaling options, particularly through Kubernetes integration, which efficiently manages deployment sizes and scales. Users can customize maps with new drawing tools for geo-annotations and set up geofencing alerts, while dashboards can be configured to update automatically based on map navigation. Additionally, Elastic Maps facilitates easier data visualization with features like no-click and one-click maps in Kibana, enhanced anomaly detection with choropleth maps, and a time slider for temporal data exploration, offering a comprehensive tool for geographic data analysis and presentation.

In an increasingly complex IT landscape, the integration of Elastic Observability and StackState enhances the ability to manage and troubleshoot dynamic environments by combining Elastic's robust data foundation with StackState's real-time topology insights. As systems grow more autonomous, the challenge lies in extracting actionable insights from vast amounts of data. Elastic provides a unified view of monitoring data, while StackState enriches this with a time-traveling topology that tracks changes and dependencies, crucial for pinpointing root causes of performance issues. This partnership eliminates the need for data duplication, leveraging Elastic's storage solutions, and allows StackState to utilize existing Elastic data to enhance problem-solving capabilities. This collaboration aims to automate root cause analysis, making it easier to navigate the complexities of modern cloud-native environments and prevent issues before they arise, while allowing users to efficiently manage data storage and compute resources.

Elastic and HashiCorp have expanded their partnership to enhance the infrastructure-as-code experience by introducing a verified Terraform provider for Elastic Cloud. This collaboration allows operations and SRE teams to manage Elastic Cloud deployments using HashiCorp Terraform across all public clouds and on-premises environments. The Terraform provider supports Elasticsearch Service on Elastic Cloud, Elastic Cloud Enterprise, and Elasticsearch Service Private environments, enabling features like autoscaling and deployment aliases. Users can dynamically scale deployments, manage deployment endpoint aliases, and utilize deployment extensions to manage plugins and script bundles. The provider also runs on ARM64 devices, allowing users to manage infrastructure from devices like the Apple Macbook with the M1 chip. Interested users can start with the provider by accessing the Terraform Registry and explore its documentation, with HashiCorp offering free remote state storage on Terraform Cloud.

De Watergroep, responsible for supplying water to over 3 million customers in Belgium, partnered with Devoteam Netherlands to enhance its IT processes and ensure high-quality customer service through the implementation of an Elastic Observability pipeline. This collaboration aimed to monitor and gain control over De Watergroep's ICT landscape, crucial for synchronizing service orders for field engineers. Devoteam, an Elastic partner, provided a centralized monitoring solution utilizing Kibana dashboards and Elasticsearch Watchers, allowing for real-time insights and automated actions to preemptively address potential issues. The monitoring system ensures that field engineers receive timely service orders, improving customer satisfaction. Additionally, the data collected is used to generate reports that help De Watergroep refine business processes and establish key performance indicators (KPIs). The partnership with Elastic allows Devoteam to scale its business and offer comprehensive solutions in observability, analytics, and security, collecting and utilizing data from various sources within organizations.

The integration of Google Cloud Private Service Connect with Elastic Cloud is now generally available, offering secure, private connections from clients in Google Cloud environments to Elastic Cloud deployment endpoints. This enhancement is accessible to customers across all subscription tiers and cloud providers, including AWS, Microsoft Azure, and Google Cloud. Private Service Connect facilitates private connectivity and policy enforcement within virtual private clouds, ensuring that all traffic remains within the Google network while being easy to set up and scale. Users must first obtain a static IP on Google Cloud to create a Private Service Connect endpoint, followed by creating a DNS record to access the endpoint with a friendly domain name. Traffic filters can be configured to allow traffic only from specific endpoints, enhancing security for Elastic Cloud deployments. Customers interested in using this feature can start through the Google Cloud Marketplace and benefit from a promotional coupon applicable to their Google Cloud billing account.

Smarter City Solutions has successfully integrated Elastic Cloud on AWS to enhance the Smarter Parking Platform, providing seamless real-time parking services across Australia and New Zealand. By utilizing Elastic Cloud's full-stack visibility and Elasticsearch for data storage, the company has improved customer experience, scalability, and operational efficiency. The implementation of Elastic's solutions, such as Application Performance Monitoring and centralized logging, has reduced the time to resolve performance issues by 50%, minimized outages, and improved data-driven decision-making. The familiarity of the engineering team with Elastic tools facilitated rapid deployment and integration, allowing the company to address issues promptly and explore further innovations like Elastic Agent and Kibana Alerting for future scalability and automation. The partnership with Elastic and AWS continues to drive growth and enhance the customer experience for Smarter City Solutions.

The Elastic Infosec Detections and Analytics team employs a comprehensive security infrastructure using Elastic's own products to collect, analyze, and act on data for security detection, incident response, threat hunting, threat intelligence, compliance auditing, and vulnerability management. The team, referred to as "Customer Zero," integrates various tools such as Auditbeat, Filebeat, and Endgame across multiple clusters to monitor and protect Elastic's systems. Auditbeat is used for monitoring Linux servers and containers, collecting data on process execution, logins, and network connections, while Filebeat gathers logs from third-party services like Okta, Office 365, and Google Workspace. Additionally, they use a Monitoring Cluster for auditing activities and a Fleet Cluster to manage Elastic Agents for data collection. A Malware Sandbox cluster allows analysts to centralize malware analysis and share findings across the team. This setup facilitates seamless cross-cluster search capabilities, enabling security analysts to access a unified interface for detecting and responding to threats. The series of blog posts provides further insights into their infrastructure and practices.

As the Army transitions from the Industrial Age to the Information Age, it seeks adaptable technologies that meet the Department of Defense's data strategy principles and the Army's data imperatives of speed, scale, and resilience. Elastic provides a powerful search platform that enhances commanders' and warfighters' capabilities by ensuring multi-domain data is accessible and actionable, enabling real-time situational awareness, and detecting adversaries across domains using machine learning. Their platform supports rapid data insights through tools like schema on write and schema on read, facilitates large-scale data visibility with cross-cluster replication and search, and maintains functionality in disconnected, intermittent, and limited environments by allowing machine learning models to run locally. This approach helps accelerate decision-making processes and equips the Army with a significant information advantage on the modern battlefield.

Version 7.14.1 of the Elastic Stack has been released, offering fixes and minor enhancements to the stack's components. A significant update in Elasticsearch addresses a memory leak issue that could impact users of Cross-Cluster Replication (CCR). Users are encouraged to upgrade to this version to benefit from these improvements. Detailed information on the changes for each product within the Elastic Stack, including Elasticsearch, Kibana, Beats, Logstash, Elastic Enterprise Search, Elastic Observability, APM, and Elastic Security, can be found in the 7.14.1 release notes.