May 2020 Summaries

The text discusses the challenges faced by data analysts in a distributed computing environment, particularly with SQL queries taking multiple hours to return results. It highlights Acceldata's approach to addressing these issues through its Query 360 tool, which provides a comprehensive view of query performance, execution stages, and data layout on the physical file system. The tool supports query plans for all flavors of Hive & Map Reduce, enabling admins and data engineers to make informed decisions about optimizing query performance.

The text discusses strategies for Hadoop admins and data engineers to improve their BI SLAs, focusing on preemptive actions that can prevent failures. It is divided into three sections: In-Flight Alerting, SLA Enforcement through Automated Actions, and Better Data Alerts. Acceldata's approach allows for in-flight, correlated data, complex combinatorial conditions across various metrics, and integration with enterprise operational run-books. The text also mentions the ability to create alerts and automated actions using their platform.

The text describes the advanced capacity warning and prediction capabilities of Acceldata's Data Observability Platform for modern data applications that require large resources such as streaming operations, ETL, and batch processing. It explains how the platform helps forecast resource outages and provides advanced warnings to cluster administrators. The solution involves creating a generic data processing pipeline using machine learning algorithms like Facebook's Prophet, LSTM, and VAR. The results are written back to a TimeSeries Database (TSDB), which can be visualized on customizable dashboards for cluster admins. This approach has helped customers manage their AWS costs dynamically by adjusting compute capacity based on accurate predictions of Yarn capacity usage.

Enterprise data infrastructure is growing rapidly in size, complexity, and business value, often relying on open source software and open core for building data-intensive applications. However, the interconnected nature of these systems can lead to operational failures if not properly managed. Acceldata's alerting platform aims to improve reliability by providing advanced notifications and alerts for various situations such as lack of capacity, increasing data partitions, SLA violations, and more. The system is designed with separation from core data systems, a unified DSL for creating alerts, robust evaluation of rules, and support for various metrics datasources. Key components include the Alert Service, Evaluators, Notification System, REST Server, and Administration Interface.