August 2019 Summaries

With the rapid evolution of malware and the limitations of traditional anti-virus techniques, machine learning has emerged as a promising tool for detecting new malware variants. This analysis explores the use of Elastic's outlier detection functionality, leveraging byte histogram profiles from binaries to identify potentially malicious software. The EMBER dataset, comprising features from 1.1 million Portable Executable files, serves as the foundation for experiments in classification and outlier detection, employing an ensemble of established outlier detection algorithms. The results indicate that byte histograms can highlight irregularities in malicious binaries compared to benign ones, particularly in the presence of obfuscation techniques that obscure ASCII characters. Although the study shows promise, it also underscores the need for further refinement in feature selection and threshold determination to enhance detection accuracy, demonstrating the potential of machine learning as a valuable asset for malware analysts.

NowPow, a Chicago-based startup, is revolutionizing patient referrals by partnering with health systems, health plans, and community-based organizations to connect people with hyperlocal resources that address social needs and health management. By integrating Elasticsearch into its operations, NowPow has significantly enhanced its search and resource matching capabilities, allowing caregivers to efficiently find and book tailored health and social services for patients. This technological upgrade has replaced outdated paper-based systems, enabling real-time search and rendering results, which has positioned NowPow as a leader in its field. With its resource directory becoming increasingly integrated into hospitals and public health departments, NowPow envisions a future where its services are widespread across the United States. This innovative approach earned the company an Elastic Cause Award, underscoring its success and commitment to improving community health care.

Elasticsearch, originally a search engine, has evolved into a highly efficient, scalable datastore that can process and retrieve time series data, which are data points associated with specific timestamps. While specialized databases exist for time series data, Elasticsearch can handle it using queries and aggregations, particularly useful in applications like server monitoring. A case study using Metricbeat, a tool that ships data to Elasticsearch, illustrates how to query time series data, with documents containing time series data and metadata. The process involves using Elasticsearch queries to filter and aggregate data efficiently, such as employing date histogram aggregations for downsampling data to a coarser granularity, which is often necessary for visualization or further processing. For specific metrics like CPU usage and network traffic, Elasticsearch's aggregation capabilities allow for detailed analysis and calculations, including downsampling and deriving rates per time unit. This functionality is facilitated by Elasticsearch's support for queries and aggregations, which can be tailored to specific time intervals and conditions, demonstrating its flexibility and power in managing time series data within a unified framework.

AMPATH (Academic Model Providing Access to Healthcare) is a collaborative initiative that addresses healthcare challenges in rural Kenya, particularly focusing on HIV treatment, by implementing digital solutions to replace inefficient paper records. Initially founded in 2001, AMPATH partnered with Moi University, the Kenyan Ministry of Health, and several American universities to create an open-source electronic medical record system called OpenMRS. By 2015, this system was enhanced with a tablet-based application, the AMPATH Medical Record System (AMRS), to streamline clinical appointments. To manage and visualize the vast amounts of patient data generated daily, AMPATH transitioned to using the Elastic Stack, which includes Elasticsearch, Logstash, and Kibana, enabling efficient data processing and visualization. This move significantly improved data management, allowing for real-time reporting and decision-making. The system facilitates monitoring and visualizing HIV-related health data, among other uses, and supports AMPATH's goal of providing timely and accessible healthcare in low-resource settings. As AMPATH continues to expand, it aims to cover new health areas and enhance user training in data visualization, leveraging Elastic's machine learning features to further improve system performance and healthcare delivery.

In a blog post by Monina Nepomuceno, the integration of Endgame's data with Elastic's suite, particularly through the use of Elastic's Canvas application, is explored. This integration follows Elastic's acquisition of Endgame in October 2019 and allows for enhanced visualization of security data. Nepomuceno, a Product Design Intern, undertook a project to design an alert dashboard using Canvas, which is notable for its customizable and dynamic dashboard capabilities. This project aimed to improve the visualization of security alerts and was showcased at the 2019 Black Hat conference. The dashboard offers managers a high-level overview of security health, with unlimited customization options to suit various user needs. The initiative underscores the potential of the collaboration between Endgame and Elastic to enhance data visualization and reporting capabilities in cybersecurity.

Elastic Maps, initially released in beta with Kibana 6.7 and now generally available in Kibana 7.3, has introduced several noteworthy features that enhance its functionality and user experience. Among these enhancements are the ability to embed maps in dashboards, facilitating the integration of geospatial data with other visualizations, and a GeoJSON upload feature that simplifies the ingestion of custom shape files into Elasticsearch. The improved legend now offers detailed insights into map layers, while users can symbolize points with customizable icons and benefit from dark-themed tiles that align with Kibana’s dark mode. Tooltips have been upgraded to display multiple features with pagination controls, and class styling now allows for refined symbolization through layer filtering. The "Most recent entities" feature leverages Elasticsearch aggregations to visualize recent document data effectively, and new subdivision layers provide more granular regional insights. Additionally, users can disable the global search bar for specific layers to prevent unnecessary data filtering, especially useful when dealing with maps that draw from multiple indices. These features collectively enhance Elastic Maps' capability as a robust tool for geospatial data analysis and visualization.

StormCrawler, an open-source web crawler written in Java, leverages Apache Storm for scalability and Elasticsearch for storage and indexing, making it both lightweight and versatile. Its modular design allows for easy extension and customization, enabling organizations to use it for large-scale data crawling and search indexing. The integration with Elasticsearch not only facilitates the indexing of crawled data, including metadata such as URLs, titles, and keywords, but also enhances performance and visibility through powerful analytics tools like Kibana. Various organizations, such as the Government of Northwestern Territory and Common Crawl, have successfully employed StormCrawler alongside Elasticsearch to replace legacy systems and maintain vast web archives, respectively. Pixray uses customized versions of StormCrawler for extensive image tracking, benefiting from the real-time data insights and responsiveness it provides compared to their previous Apache Nutch setup. Julien Nioche, the author of StormCrawler, highlights its continuous improvements and potential future enhancements, such as integrating the new Storm metrics API with Elasticsearch for broader applicability beyond web crawling.

Candid, formed from the merger of GuideStar and Foundation Center, serves as a crucial resource for nonprofit information, with over 10 million annual users and extensive data partnerships. The organization aims to connect people with the information needed to effect positive change by maintaining profiles of over 2.7 million U.S. charities, drawing on data from the IRS and direct nonprofit contributions. Central to Candid's mission is its search engine, powered by Elasticsearch, which enhances the search experience and was recognized with an Elastic Cause Award. According to Shane Ward, Senior Director of Technology, Elasticsearch is integral to Candid's operations, having replaced their previous technology and becoming a core part of their API platforms. Candid plans to further utilize the Elastic Stack to expand its capabilities, including enabling volunteer opportunity searches, creating network diagrams, and developing map-based search functionality to offer a comprehensive view of the social sector.

Perched, a company specializing in security analytics, threat hunting, and security operations, has joined Elastic to provide enhanced training and consulting services for cybersecurity solutions utilizing the Elastic Stack. This collaboration will leverage Perched's deep expertise in SIEM, threat hunting, network security monitoring, and building threat models, along with their experience as long-time Elastic Stack users and partners. The Perched team has contributed significantly to the development of Elastic's security products, including extending the Elastic Common Schema and providing input for the new Elastic SIEM product. The partnership aims to offer unmatched security expertise to Elastic’s customers, continuing with independent open-source projects like RockNSM and CAPESstack. The founders of Perched, Andrew Pease and Derek Ditch, express enthusiasm about the merger, highlighting their commitment to operationalizing the Elastic Stack for cybersecurity and scaling analysis with open-source technology.

Synonyms play a crucial role in enhancing the capabilities of search systems, including Elasticsearch, by allowing different words with similar meanings to be treated equally during searches. Synonym filters can be applied at both index time and search time, each with distinct advantages and challenges. While index-time synonyms may lead to increased index size and require reindexing to update rules, search-time synonyms maintain original index size and allow for dynamic updates without reindexing. Recent improvements in Elasticsearch, particularly the introduction of the synonyms API in version 8.10, enable real-time updates of search-time analyzers, facilitating more flexible synonym management. Properly leveraging synonyms can improve search recall and relevance, but it requires careful consideration of domain-specific language and potential impacts on search scoring. The blog emphasizes the importance of ongoing experimentation and testing to maximize the benefits of synonyms while minimizing unexpected side effects.

Elastic Certified Engineer certification has significantly benefited professionals in various roles, leading to career advancements, increased job opportunities, and enhanced technical skills in using Elasticsearch. Individuals such as Jessé Peixoto, Lucian Precup, Joe Wu, Vojtěch Hýža, and Mehmet Tolga Avcioglu have experienced wage increases, new responsibilities, and recognition in their fields after obtaining the certification. The rigorous certification process involves practical tasks on Elasticsearch clusters, which helps candidates deepen their understanding and application of Elastic technologies. This credential has helped these professionals better serve their clients, expand their career opportunities, and contribute to their companies' success by utilizing advanced features of Elasticsearch. The certification's growing demand indicates its value in demonstrating expertise in Elastic Stack technologies, and the introduction of new certifications, such as the Elastic Certified Analyst, reflects the evolving landscape of data analytics and observation.

SolveBio, a New York-based startup, operates at the intersection of technology and genomics to advance precision medicine by aiding pharmaceutical companies in oncology research and development. The company leverages the Elastic Stack as a crucial component of its molecular data platform, enabling the aggregation and analysis of complex genomic data for drug discovery. This platform allows scientists to store and scrutinize semi-structured data from a myriad of sources, including molecular diagnostics reports and clinical studies, enhancing their ability to identify genetic mutations and understand clinical trial outcomes. SolveBio facilitates targeted drug development by bringing scientists closer to their data, allowing them to pose complex questions without the need for a bioinformatics expert. By integrating Elasticsearch, SolveBio has enhanced its platform's capacity to handle diverse molecular datasets, supporting adaptive clinical trials and the discovery of new therapies. The company's innovative use of Elasticsearch has been recognized with a Cluster Award for technology innovation, underscoring its significant contribution to the field of precision medicine.

The 2019 Elastic Search Awards for the Asia Pacific and Japan (APJ) region celebrated innovative uses of the Elastic Stack, with honorees including GirlScript Foundation, InfoTrack, Pathfinder Labs, Sumitomo Mitsui DS Asset Management, The Warehouse Group, and Zero Latency VR. These organizations demonstrated how Elastic's search technology can be adapted for diverse applications, from humanitarian efforts to business transformations. The awards were divided into three categories: Cause Awards for projects that better humanity, Cluster Awards for technological innovation, and You Know, for Search! Awards for business transformation. Notable projects included Pathfinder Labs' ODIN tool for combating child exploitation, GirlScript Foundation's initiatives to support budding technologists, and Sumitomo Mitsui's ALPHA system for real-time asset management decisions. Additionally, Zero Latency VR utilized Elastic for real-time infrastructure monitoring, The Warehouse Group improved their returns process with a custom search solution, and InfoTrack developed a data lake for enhanced search capabilities. The awards, presented during Elastic{ON} in Sydney, underscored the impact and potential of Elastic Stack in fostering innovation across various sectors.

Samantha Menot shares her journey of personal and professional growth, influenced by her multicultural upbringing and inspired by her grandfather's work-life balance. She recounts her move to London to rediscover her European roots and her subsequent career at Elastic, where she found a supportive environment that aligns with her passions for people, running, and community service. At Elastic, she has been instrumental in building user success teams, participating in diverse volunteer initiatives like The Salvation Army Christmas Present Appeal and the Brixton Soup Kitchen, and leading efforts in various charitable activities. Her story highlights the impact of following one's passions and the importance of a workplace that fosters personal development and community involvement, culminating in her participation in the Boston Marathon to support the charity Back on My Feet. Through these experiences, Samantha expresses gratitude for the opportunities to grow and give back, emphasizing how Elastic has enabled her to pursue a fulfilling life both professionally and personally.

Seccomp, short for secure computing mode, is a Linux kernel feature that restricts the system calls a process can make, enhancing security by preventing unauthorized code execution. It is particularly useful for mitigating remote code execution vulnerabilities that can lead to further malicious activities. Elasticsearch integrates seccomp to enhance security by installing system call filters early in its startup process, using Java Native Access (JNA) to access native libraries. This process blocks potentially harmful system calls like FORK, VFORK, EXECVE, and EXECVEAT, which are essential for starting new processes. Seccomp is also employed in Beats, a component of the Elastic Stack, where it is configured via YAML to allow or reject specific system calls, using a whitelist approach to enhance security further. The blog post also touches on the use of Auditbeat for logging seccomp violations and demonstrates how to generate seccomp events with tools like firejail, allowing users to monitor and adapt their security policies over time. The Elastic Stack's use of seccomp underscores the importance of leveraging existing operating system features to enhance security, rather than attempting to develop custom solutions.

The blog post by Yukiya Shimizu explores how machine learning (ML) features in the Elastic Stack can be utilized to analyze data from Meetup, a platform for organizing in-person events based on shared interests. The Elastic Stack employs unsupervised learning to detect anomalies in time series data, such as unexpected spikes in the number of Meetup groups created. By ingesting data from the Meetup API into Elasticsearch, the blog demonstrates how ML jobs can identify anomalies, such as the creation of 848 groups in a single day, which was linked to a TechCrunch article about Meetup seeding 1,000 #Resist groups. The post also provides insights on configuring ML jobs, highlighting the importance of setting appropriate time windows and handling data noise to improve the accuracy of anomaly detection. The capabilities of Elastic Stack in uncovering significant patterns in large datasets without requiring users to be data scientists are emphasized, encouraging users to explore its ML features for new discoveries.

Cross-cluster search (CCS) is now generally available on the Elasticsearch Service on Elastic Cloud, allowing users to search across multiple deployments from a single endpoint and Kibana instance. Initially, this feature supports federated searches within the same region for Elastic Stack 6.7+ deployments, with plans to expand across regions in future updates. The Elasticsearch Service simplifies the management of multiple deployments, enabling better isolation, backup control, and upgrade scheduling. CCS enhances logging and observability by allowing separate deployments per log and data type, leading to improved performance and scalability. It also supports searching across different major versions of Elasticsearch, providing flexibility for teams to upgrade at their own pace. Additional features in this release include OpenID Connect single sign-on and Kerberos authentication for securing clusters and instances, with a 14-day free trial available for new users.

Vectorspace AI utilizes Natural Language Understanding (NLU) and the Elastic Stack to create and visualize datasets that uncover hidden relationships between genes, stocks, and other entities in life sciences and financial markets. Initially developed at Lawrence Berkeley National Laboratory, these datasets leverage feature vectors and word embeddings to analyze correlations in data sources such as scientific literature and genomic databases. By augmenting financial datasets with these vectors, Vectorspace can generate alpha by predicting price movements in stocks, demonstrated through events involving companies like Merck and Celgene. The approach draws parallels between the behavior of stocks and genes, using NLU-based correlations to identify latent interconnections that can be visualized with tools like Kibana's Canvas. This methodology not only aids in financial markets by optimizing signal-to-noise ratios and generating actionable insights but also supports research in areas like human spaceflight safety. Through collaborations with institutions such as NASA, DARPA, and Genentech, Vectorspace continues to expand its applications in both scientific discovery and market analysis.

Elastic Enterprise Search Beta 2 introduces enhanced connectivity and extensibility features, allowing organizations to efficiently search across various internal and cloud-based applications through easy-to-use connectors and custom API sources. This release builds on the initial beta by incorporating custom schemas for unique data representation and offering new connectors for popular tools like JIRA and Confluence, facilitating better collaboration and faster execution within teams. It emphasizes the flexibility to ingest diverse data types without rigid schemas, enabling developers to map specific content characteristics. The update aims to streamline search implementations and improve productivity by simplifying the integration of enterprise systems. Elastic invites feedback from users to refine the product further, acknowledging that while this beta provides valuable insights, it is not yet suitable for production environments.

Elastic, a company committed to community engagement, offers its employees volunteer time off (VTO) to support various charitable activities. In a featured edition of their Helper Node blog series, Nick Wilkinson, a sales development representative at Elastic's Sydney office, shares his experience volunteering with Habitat for Humanity in Yogyakarta, Indonesia. Wilkinson and his team helped build a house by performing tasks like digging foundations and laying bricks, collaborating with local villagers who demonstrated a strong sense of community and cooperation. The project was not only about physical labor but also involved meeting a fundraising goal to support ongoing Habitat projects. Despite work and financial constraints, the supportive environment at Elastic, combined with the structured assistance from Habitat for Humanity, facilitated a smooth volunteering experience. Wilkinson emphasizes the profound impact and fulfillment gained from such volunteering efforts and encourages others to participate in similar initiatives.

Metricbeat has introduced a new concept called "light modules," which are predefined configurations for generic metricsets that do not require Go code and are designed to facilitate the monitoring of services using existing protocols supported by Metricbeat. These light modules allow users to extend Metricbeat functionality by providing a way to share configurations and dashboards, making it easier and faster to develop new modules for services that can be monitored using existing metricsets. A notable example is the light module for CockroachDB, which uses the Prometheus collector metricset but customizes the metrics endpoint. Light modules are structured with metadata files such as module.yml and manifest.yml, and they can include documentation, field mappings, and dashboards. They serve as a fallback mechanism, where Metricbeat will load a light module if a native module is not found in the configuration. This approach is already being applied to services like AWS ELB and is expected to expand to more services, with ongoing work to improve tooling and support for contributors in developing these modules.

At the Tochigi Ruby Conference, a presentation was given on monitoring Ruby on Rails applications using the Elastic Stack, focusing on key areas such as uptime monitoring, host and application metrics, log monitoring, and application performance monitoring (APM). The demonstration involved a Rails application with a frontend composed of NGINX and Rails and a backend using PostgreSQL, incorporating tools like Metricbeat, Filebeat, Auditbeat, Packetbeat, Heartbeat, and the Elastic APM Ruby Agent to gather comprehensive monitoring data. Elasticsearch Service was used to streamline the setup for data storage and visualization, allowing for easy monitoring of server and application availability through the Uptime app in Kibana. The presentation emphasized the importance of gathering detailed logs for problem identification and the role of Elastic APM in measuring application performance internally, highlighting the emerging concept of observability, which enables the correlation of metrics, logs, and APM data in a unified platform to expedite problem-solving. The Elastic Stack facilitates this observability, offering a trial for users to explore its capabilities in monitoring and data visualization.

The Kibana UI Weekly Update for August 2nd, 2019, highlights significant progress across various facets of the platform, including UI enhancements, security features, and platform operations. Key developments include the merging of the new route handler interface for the HttpService, enhancements to the AppService and ContextService, and advancements in the New Platform Migration with TypeScript updates and Angular cleanup. The Lens visual editor saw performance improvements, chart-switching capabilities, and better suggestion logic, while the Discover application introduced multi-column sorting. The EUI Data Grid component made strides in accessibility, pagination, and initial styling, with a focus on delivering it by month-end. Additionally, the Security plugin gained the ability to create API keys, facilitating requests in alerting processes. The update also notes ongoing recruitment for various Kibana engineering positions.

Cytech Mobile, a technology provider in the telecoms sector, leveraged the Elastic Stack to develop a business intelligence tool called mCore Dashboards within six months, aimed at enhancing their flagship cloud-based software, mCore, which manages Application-to-Person (A2P) SMS traffic. The company initially faced challenges with existing solutions that were not scalable or cost-effective for their customers. By adopting the Elastic Stack, Cytech was able to integrate features like fast full-text search, alerting, and reporting, all essential for handling over a billion transactions annually across ten countries. The transition to Elastic Cloud enabled them to align with their SaaS pricing model and offer essential user management and reporting features. Positive customer feedback highlighted the tool's efficiency in providing automated reports and alerts, saving time and money. Cytech plans to expand mCore Dashboards with Elastic's machine learning capabilities and Canvas feature, aiming to present a new standalone product at the Mobile World Congress 2020.

Elastic is actively participating in major cybersecurity events such as Black Hat, BSidesLV, and DEF CON, emphasizing its integral role in the security community through sponsorships, trainings, and original research. The Elastic Stack, widely utilized in security operations centers for its speed, scale, and relevance, is a key component in many organizations' security analytics and SIEM solutions. Elastic recently launched its own SIEM in Elastic Stack 7.2 as part of its free Basic license, which has been rapidly adopted by the security community. During these events, Elastic is hosting sessions and workshops featuring its security experts, including presentations on machine learning-based threat hunting and hands-on sessions using the Mitre ATT&CK framework. The company will also hold private workshops with partners, focusing on topics like threat hunting and the integration of SIEM with AI, underscoring its commitment to fostering collaboration and innovation in cybersecurity.