March 2018 Summaries

Rashid Khan, the creator of Kibana, presented a preview of Canvas at the Elastic{ON} 2018 conference, demonstrating its capability to create custom infographic-inspired visualizations using SQL. Canvas is described as a flexible and creative platform for live data visualization, allowing users to connect elements to data and configure them through a simple UI. During his demo, Rashid used data from slot car races to show various driver performances by creating dynamic charts with SQL. He illustrated how maximum speed does not always equate to the fastest lap completion, emphasizing the importance of balancing pace and speed. The session also highlighted upcoming features in Kibana, including improvements in accessibility, user interfaces, and solutions for logging and metrics deployment, alongside exploring geo-capabilities in Elasticsearch and Kibana.

Tony Sleva's blog post, part of the Elastic{ON} 2018 series, discusses the introduction of rollup functionality in Elasticsearch, aimed at optimizing data storage without sacrificing search speed. Rollups, which aggregate data into a new, smaller index for long-term storage, address the challenge of managing extensive time-series data in dashboards while conserving disk space and IT budgets. The new feature allows users to select fields for rollup and seamlessly query both live and rollup data using the Rollup API. This addition enhances Elasticsearch's scalability and adaptability, providing a practical solution for maintaining high-level dashboards with historical data. The post also hints at further advancements discussed at the Elastic{ON} conference, including machine learning, security analytics, and new features for Elasticsearch and Kibana.

The blog post explores the implementation of a two-level Sankey graph to visualize network traffic patterns using Vega in Kibana 6.2. A Sankey diagram, which illustrates the flow quantity between nodes with varying line widths, is created using Elasticsearch and Kibana, along with sample data generated by the makelogs utility. The visualization allows users to view all traffic or filter it by selecting specific source or destination countries. The post outlines the process of setting up the environment, generating data, and crafting the visualization using various data tables and transforms in Vega, such as nodes, edges, and groups. It emphasizes user interactions through signals for dynamic filtering and highlights debugging strategies using browser tools. The post also provides insights into manipulating Vega code to customize the graph's appearance and interactions, offering a detailed walkthrough of the technical steps involved in creating this visualization.

At the Elastic{ON} 2018 conference, the Ask Me Anything (AMA) Booth was a bustling hub where users, customers, and partners engaged with Elasticians to discuss products, processes, and use cases. The booth was staffed by a diverse team, including support engineers, consultants, product managers, and developers, who used tools like dry-erase markers and notebooks to facilitate in-depth conversations. This event also featured the Canvas workpads, a technology preview offering a creative space for live data visualization, showcased through various configurations and visual elements. The workpads included a range of dynamic visual components such as group bubbles, image fill elements, a dynamic bar chart, and a consultation time series, all designed to display data gathered from an internal survey app in real-time. The data was processed using Elasticsearch and visualized using different techniques and tools within the Canvas platform, demonstrating the capabilities of Canvas in presenting complex data in an engaging and understandable format.

The article explores the implementation of pseudonymization techniques for protecting Personal Data as defined by the General Data Protection Regulation (GDPR), focusing primarily on data ingestion into Elasticsearch using Logstash. Pseudonymization is highlighted as a GDPR principle for securing Personal Data, allowing organizations to maintain operational capabilities while reducing protection costs compared to leaving data intact. The article distinguishes pseudonymous data from anonymous data, emphasizing that pseudonymized data still allows for potential re-identification if additional information is available, thereby requiring appropriate security measures. It details the use of hashing methods, such as SHA256, and key management strategies to create and store pseudonymized data in a secure identity store, ensuring that original data can be re-associated when necessary. The article also cautions about the potential risks of hash reversal and the importance of managing keys and secrets securely, suggesting periodic key rotation to bolster security. Future discussions are anticipated to cover the reversal of pseudonymized data at scale and the tagging of Personal Data during and after ingestion, with a focus on maintaining data utility for analysis while complying with GDPR requirements.

Five Big Ten universities, led by Indiana University, have established OmniSOC, a cybersecurity operations center leveraging the Elastic Stack to enhance threat detection and response times across their campuses. This collaborative initiative, involving Indiana University, Northwestern University, Purdue University, Rutgers University, and the University of Nebraska-Lincoln, aims to protect students, faculty, and staff from cyber attacks by sharing threat intelligence and using advanced analytics. OmniSOC operates using components of the Elastic Stack, such as Elasticsearch, Kibana, Beats, and Logstash, to ingest and analyze vast amounts of data from thousands of systems, enabling rapid identification and mitigation of cyber threats. The initiative reflects a broader trend of using data-driven approaches and predictive analytics to enhance cybersecurity, with plans to expand its services and membership beyond the Big Ten Academic Alliance. The use of the Elastic Stack not only provides deep visibility into security information but also promises future applications, such as a multi-petabyte digital archive and additional operational logging use cases.

In March 2018, Endgame introduced Red Team Automation (RTA), a framework designed to help organizations validate their security defenses against adversarial behaviors outlined in the MITRE ATT&CK™ matrix. The RTA toolset consists of 38 scripts and executables that simulate various techniques used by adversaries, allowing security teams to assess and improve their detection capabilities. Unlike other adversary simulation tools, RTA is noted for its simplicity and ease of extension, providing a low-overhead solution for testing security measures. The framework currently covers over a quarter of the ATT&CK™ matrix and is intended to grow with community contributions. It supports the broader open-source community by offering a platform for collaborative development and enhancement of defensive capabilities. RTA scripts are designed to simulate behaviors like unusual network activity, PowerShell abuse, and persistence mechanisms, offering a practical way for organizations to measure and improve their detection coverage. By releasing RTA as an open-source tool, Endgame aims to foster community engagement and provide a resource for organizations to better understand and respond to emerging threats.

At the Elastic{ON} 2018 conference, Ryan Kazanciyan, the technical consultant for USA Network’s Mr. Robot, detailed how the show incorporates realistic security events using tools like the Elastic Stack and Kibana. He explained how the show's protagonist, Elliott, used these technologies to track the fictional Dark Army, emphasizing the commitment to authenticity by basing scenarios on actual security threats and ensuring accurate technical details. The Mr. Robot team is known for their meticulous attention to detail, drawing inspiration from real-world attacks and carefully crafting each on-screen element. Kazanciyan highlighted the show's focus on realistic depictions of cybersecurity, which has resonated strongly with the Elastic community, evidenced by significant positive feedback. The session also explored how visualizations in Kibana aid in storytelling and threat detection, reflecting the broader theme of the conference, which covered various applications of the Elastic Stack, including machine learning, security analytics, and operational data analysis.

Canvas, a technology preview from Elastic, offers a dynamic and customizable space for presenting live data in a visually appealing way, complementing Kibana dashboards. At Elastic{ON} 2018, a creative project titled "Elasticoffee" showcased the potential of Canvas through a mockup café workpad that tracked coffee consumption using Z-Wave buttons and a Raspberry Pi 3 to collect data sent to an Elastic cluster. The workpad featured a chalkboard style with hand-drawn elements, displaying various beverage options and metrics through interactive components like bar graphs and image reveals. The project highlighted Canvas's flexibility in data visualization, allowing users to express data creatively while learning the power of its code editor for customization. Future developments aim to integrate Canvas into X-Pack, enhancing its functionality and accessibility.

The blog post introduces the roadmap for geo features in Elasticsearch and Kibana, highlighting enhancements that allow users to create multi-layered maps with advanced geospatial data analysis capabilities. Users will be able to choose base layers from the Elastic Maps Service, add multiple data layers, and customize them through stylesheets and shape layers, with no limits on the number of layers. The integration of GeoJSON directly into the Kibana UI will automatically create indices in Elasticsearch, facilitating seamless use across different maps. Additional features in development include support for Vega visualizations, custom map layers, on-premises map deployment, and region maps, all aimed at expanding the geospatial analytical power of the Elastic Stack. The post is part of a series from the Elastic{ON} 2018 conference, which also covered various other topics related to Elasticsearch, such as machine learning forecasting, security analytics, and operational data analysis.

Albert Zaharovits discusses best practices for developing plugins to work alongside the X-Pack Security Plugin for Elasticsearch, emphasizing the importance of adhering to established authentication and authorization frameworks. As there is no stable plugin API in Elasticsearch, developers should consider the necessity of custom plugins, especially since many features are now part of the core Elastic Stack. The Security plugin provides various features such as inter-node secure communication, HTTPS, and auditing, but it does not automatically detect invalid access patterns by other plugins. Authors should focus on implementing TransportAction for custom requests, using a distinct naming convention for actions to ensure they align with privileges, and delegating execution properly. It is crucial not to splice authentication onto actions or manage users and roles, as these are handled by the Security plugin. Developers should use ThreadContext and TransportService to maintain thread safety and rely on NodeClient for executing actions, ensuring that all actions are named hierarchically to facilitate authorization through predefined role privileges.

International Women's Day (IWD) serves as a global celebration of women's achievements and a call for gender parity, themes strongly embraced by Elastic, a technology company striving to enhance gender diversity and inclusion within its workforce. Elastic has made concerted efforts to recruit more women, resulting in a 6% rise in female employees, alongside initiatives to create a supportive workplace that values kindness and compassion. The company has introduced 16 weeks of paid parental leave for all parents, emphasizing the importance of equal opportunities at work and home. Through events like the Women’s Breakfast at their Elastic{ON} conference, Elastic fosters dialogue on diversity and inclusion, aiming to extend these discussions globally. Despite acknowledging the long road to gender parity, Elastic remains committed to making meaningful changes, believing in the impact of collective efforts to foster an equitable environment.