October 2021 Summaries

The text outlines a comprehensive approach to observability and data health checks within data pipelines, focusing on the integration of Apache Druid with systems like Apache Airflow, Opsgenie, and Slack. It emphasizes the importance of maintaining data quality and completeness from the initial stages of raw data processing through to analysis, using both static rule checks and dynamic, data-driven tests to ensure accuracy and reliability. The article discusses the trade-offs between cost, timeliness, and data validation, advocating for an iterative approach to testing, starting small and expanding as more is learned from production pipelines. It highlights the need to identify root causes of pipeline ingestion failures quickly and to automate responses where possible to minimize data lag. Monitoring end-user performance is also crucial, particularly for optimizing query latency on massive datasets, with the use of Rill Explore dashboards to diagnose issues. The text concludes by suggesting the inclusion of business stakeholders in the alerting process and conducting post-mortems to refine workflows and reduce future issues, sharing insights gained from their journey in maintaining an always-on observability system.

Maintaining mission-critical services at scale requires robust operational analytics systems with high availability and low data latency. At Rill, this involves proactive monitoring of data pipelines and infrastructure using tools like Apache Druid and Airflow. The approach emphasizes early warnings, integration into existing workflows, and minimal maintenance. Key strategies include using Slack and Opsgenie for alert notifications, prioritizing alerts based on urgency, and ensuring that resolution processes are linked to the teams most familiar with specific business logic. The architecture supports flexibility and comprehensive alerting through a framework that considers data lifecycle and types of alerts. To achieve this, Rill integrates Airflow with Slack and Opsgenie, establishes priority routing for alerts, and documents incidents for learning and improvement.

Rill Data assists companies in overcoming the challenges of analyzing streaming data by providing a platform that combines efficient data modeling and high-performance processing, enabling real-time sub-second online analytical processing (OLAP). The platform integrates with BI tools like Tableau and Looker, and when data enrichment is necessary, it pairs well with ksqlDB and Apache Druid, facilitating the transformation of streaming data into actionable insights. An illustrative exercise using OpenSky Network aircraft telemetry data demonstrates the ease of setting up data streams through Rill, enriching them with ksqlDB user-defined functions, and capturing data for OLAP queries using Apache Druid. Visualization tools such as Apache Superset are used to display the enriched data, providing dynamic, real-time insights into aircraft movements. The text also highlights the importance of preventing data duplication and ensuring exactly-once semantics within streaming applications, with Rill managing Apache Druid and real-time Kafka infrastructure for clients. The open-source tools and scripts supporting these processes are available in a GitHub repository, offering a comprehensive demonstration and exploration of the technologies involved.

The "Guide to Apache Druid" serves as an evolving resource for individuals new to the data analytics platform, offering a curated list of articles, customer stories, and architectural diagrams to expedite learning about its application and integration within the Modern Data Stack. It highlights how major companies like Netflix, Airbnb, and Salesforce utilize Apache Druid for real-time analytics, detailing its role alongside other databases such as Clickhouse and Pinot in handling large-scale data processing. By exploring customer narratives and reference architecture diagrams, the guide illustrates the diverse use cases and advantages of employing Apache Druid in conjunction with other data storage solutions. The guide underscores the widespread practice of "polyDBism," where real-time databases are used complementarily with data lakes, warehouses, and other data management technologies, emphasizing the platform's scalability and flexibility.

Apache Druid, a real-time analytics database, is integrated with Kubernetes through a Druid Operator, which simplifies the operational management of Druid clusters by automating tasks like autoscaling, rolling upgrades, and resource cleanup. Initially, Helm Charts were used to deploy Druid clusters, but their limitations led to the development of custom Golang-based operators and Kubernetes Custom Resource Definitions (CRDs) to manage the complex system. The Druid Operator, introduced in late 2019, understands Druid's internal architecture, facilitating better uptime, high availability, and seamless rolling upgrades without downtime. It supports both StatefulSets and deployments for different node types, and automates tasks like PVC cleanup using Kubernetes finalizers. Features such as selective node upgrades and self-healing StatefulSets enhance operational efficiency, while the operator runs in a high-availability mode with a single active controller managing events. Additionally, the Kubectl Druid plugin extends Kubernetes command-line interface capabilities, simplifying Druid cluster management further.