December 2024 Summaries

Our favorite Metaplane features from 2024—and what’s coming next` The Metaplane team had a successful year in 2024, shipping 31 new features that enhanced data observability, making it more comprehensive, intuitive, and impactful. Key highlights include the launch of Data Insights, end-to-end dbt observability, intelligent incident management, source-to-target monitoring, and a native Snowflake app. The team also strengthened partnerships with Snowflake and other companies, and received significant investments to continue building their data observability tool. Looking ahead to 2025, Metaplane aims to provide a true mission control for metadata, bring data quality to the business, and help teams fix issues rather than just finding them.

dbt testing can be complex, but it's a crucial step in ensuring the accuracy and consistency of your data pipeline. Understanding the different types of dbt tests is key to making informed decisions about which tests to use and when. Data tests focus on validating the accuracy, consistency, and completeness of your data, while unit tests validate the logic of your models or macros. Generic tests are pre-defined, reusable checks that can be applied directly in your schema.yml file, while custom tests allow for more specific validation logic. dbt test packages can help extend dbt's functionality with ready-made solutions for common data quality checks and workflow improvements. By leveraging a mix of these different types of tests, you can build trust in your data and ensure the reliability of your pipeline. As your data engineering toolkit evolves, don't forget to tap into the power of pre-built packages and explore custom and unit tests as needed.

We have shipped several improvements to the accuracy and consistency of our anomaly detection, including using more than one year of history to better detect and adjust for monthly seasonality. Now, users can specify if a monitor should only have an upper or lower bound, allowing them to focus on specific metrics such as product returns versus costs. Additionally, Metaplane will no longer consider zero values in future predictions if they are never observed, and it will ensure that negative values are not part of the prediction range for certain monitors like warehouse costs or file sizes. These improvements aim to enhance data quality and observability.

Gitlab support for CI/CD capabilities has been announced, expanding proactive data quality checks to teams using GitLab repositories for dbt development workflow. This integration brings automated data quality checks and improved forecasting tools to help teams catch potential issues before they reach production. The new feature allows users to preview data impact, test changes, and be proactive about data quality, all within the merge requests in GitLab.

dbt Cloud notifications can be configured for job-level or model-level notifications, but as of December 9th, 2024, model-level notifications are only email-supported and have limitations such as sending alerts to a single email address and not supporting routing based on the event type. In contrast, Metaplane's dbt alerting tool provides more context-rich alerts with granular routing capabilities, allowing users to assess the situation immediately and dispatch alerts to the right person or team. To set up Metaplane's alerting tool, users need to connect their dbt instance and configure alert rules based on specific nodes of interest, such as tags or locations within the warehouse. The tool also allows users to send alerts to Slack or Microsoft Teams and can be integrated into a CI pipeline for real-time feedback. By using Metaplane's dbt alerting tool, users can resolve dbt failures faster and improve data quality and trust.

dbt Cloud` notifications allow configuring `job-level` and `model-level` notifications, with limitations on model-level notifications including only being sent via email and having a single owner for the model. In contrast, `Metaplane's dbt alerting tool` offers more context-rich alerts and granular routing options, allowing for better assessment of the situation and ensuring that each alert finds the right person or team to address it. To set up Metaplane's dbt alerting tool, users must create a free account, connect their `dbt Cloud` or `dbt Core` instance, configure alert rules based on specific nodes of interest, and update their `dbt project` with the new exposure. This setup allows for faster resolution of `dbt job failures`, increasing overall quality and trust in data by providing more timely and relevant alerts.

Introducing source-to-target monitoring: Keep your data consistent throughout your pipeline` Source-to-target monitoring is a new feature in Metaplane that ensures data consistency between different data sources and target destinations. Most data quality issues start at the source, but with multiple data sources, it can be challenging to maintain data consistency. This new tool allows users to compare data from multiple sources in one monitor, ensuring data stays consistent throughout the pipeline. With automated validation of data at every step of its journey, users can define a source metric, target metric, and comparison rule to ensure data consistency with minimal effort. The monitors can be configured for various scenarios, including row counts, freshness, cardinality, and group-by comparisons, using Metaplane's machine learning-powered anomaly detection. Setting up a source-to-target monitor can be done in minutes, and users can use the no-code interface or manual SQL configuration options.

The dbt alerting tool was built to provide more granular and context-rich alerts for dbt users, allowing them to configure custom alert rules without modifying their dbt projects. The tool operates under the hood using two key artifacts: `manifest.json` and `run_result.json`, which contain metadata about dbt resources and execution results, respectively. The system leverages dbt exposures to store custom configuration data and defines alert rule structures with filters and destinations, allowing users to specify conditions for matching nodes and destinations for notifications. When a dbt job runs, the system extracts alert rules from `manifest.json`, applies them to `run_result.json`, merges relevant data, generates detailed alerts, and distributes them to specified destinations. The tool also handles model-test relationships by sending alerts for both tests and models when necessary. By providing more context-rich alerts, the tool aims to enable users to diagnose issues directly from the alert itself, making it a powerful and free resource for all dbt users.

dbt model run timeline view is now available for dbt Core and dbt Cloud users, allowing them to track job performance with an interactive timeline view. This feature provides detailed execution sequences, start times, and durations to help identify bottlenecks and optimize performance. Users can zoom in or out to diagnose specific issues or spot broader patterns, prioritize optimization work on slow models, and monitor performance over time to guide data pipeline improvements. The new feature is accessible through the dbt integration dashboard.

Metaplane, a data observability company, built its own alerting infrastructure for their dbt platform to improve the actionability and context of alerts. They found that out-of-the-box dbt notifications lacked sufficient context and led to alert fatigue. To address this issue, they developed an approach that includes more context in alerts, configures alerts as code using dbt YAML files, and provides access to failing data within Snowflake. The new infrastructure has resulted in more effective triaging and fixing of critical failures.