March 2024 Summaries

In March 2024, percentage monitors will undergo enhancements to improve their accuracy in detecting changes. Nullness, uniqueness, and percent zero/negative monitors will experience these improvements over the coming weeks. The expected ranges of these monitors may change as a result. Handling of extreme values like 0% and 100% will be improved by excluding them from the expected range unless they were observed during training or marked as normal in the monitor's history. Additionally, ranges will become smaller to alert users on relatively large changes even if they are not significant on an absolute scale. The ranges will continue to shrink as more new values are observed and model confidence increases.

Metaplane has introduced a new feature called Data Insights that helps users identify opportunities for performance and cost optimization in their queries. The feature provides an overview of query costs over time, highlights the sources of these costs at both query and user levels, and pinpoints the least performant queries based on runtime. Users can explore each insight further through drill-down views, filtering, grouping, and aggregating data to optimize their queries for immediate and future gains. The Query Explorer allows users to examine specific instances of each query, while table-level insights provide information about query volume, frequent users and roles, slowest queries, and more. Metaplane customers have already found value in the Data Insights dashboard, and updates are planned to include a roll-up of unused tables and views for cleanup purposes.

A new feature has been introduced for fine-tuned regression testing in dbt CI checks. This native GitHub application automates CI checks alongside pull requests, including impact analyses and test previews to help teams merge changes with confidence. To optimize speed and power, configuration options have been refined, allowing users to filter by Metaplane tags, turn off checks, ignore draft PRs, specify where tests should run, and more. Users can get started with Metaplane for data regression testing by connecting their dbt instance and GitHub repo hosting their dbt code or creating an account.

Metaplane has updated its Looker integration to include lineage for View and Explore fields, allowing users to understand the full impact of data quality incidents on their analytics. The new features provide field-level lineage for Views, Explores, Dashboards, native (LookML) derived tables, and SQL-derived tables. By integrating Looker with Metaplane, users can proactively alert stakeholders about potential issues and accelerate root cause analysis. Support for derived tables has also been added, providing further granularity in lineage graphs. Users are encouraged to provide feedback on the integration, particularly regarding Persistent Derived Tables (PDT).

Metaplane has expanded its Databricks integration to support Hive Metastore, in addition to Unity Catalog. This allows users to leverage Metaplane's data observability platform across all their Databricks workspaces, including those using Hive Metastore-based setups. The integration enables automatic anomaly detection and powerful configuration options for identifying data quality incidents. By integrating with the rest of the data stack, users can quickly identify incident origins and prevent future issues through regression tests in their dbt CI process. Improved data quality leads to better business decisions, product development, and service provision. Metaplane also supports data observability for various data warehouses and transactional databases. New users can create an account or schedule a consultation to learn more about data observability best practices.

Metaplane has integrated with Segment, allowing for greater visibility into how data lands in your warehouse. This integration benefits larger teams by identifying upstream lineage and helping to resolve data quality incidents faster. Understanding these incidents can help address issues such as misfiring pixels or new event parameters that may result in schema changes. Metaplane also supports other upstream lineage objects, including everything in the warehouse and Fivetran. To get started with Metaplane and Segment, users need an API key to access their Segment account or can create a new account for data observability implementation within an hour.

Managing data in Snowflake is crucial for maintaining trust and efficiency within an organization. Tags are schema-level objects that can be assigned to other objects as arbitrary string values, providing a means of organizing and categorizing data. They enable easier tracking of sensitive data, resource usage, and object relations. A proper tagging strategy should include a list of objects or datasets needing tags, a tag naming convention, and use cases for assigning specific tags. Both centralized and decentralized approaches to tag management have their pros and cons, so the right approach depends on an organization's needs. To create and manage tags in Snowflake, SQL commands are used to create tags, assign them to objects, and query objects by tags. Tag quotas should also be considered when managing tags.

Metaplane, a company focused on ensuring trust in data for businesses, has raised $13.8 million in its Series A funding round led by Felicis Ventures. The investment brings the total amount raised to $22.2 million and will be used to automate the detection of data issues. Since its last fundraise, over 100 companies have trusted Metaplane for ensuring trust in their data. The company aims to help 10,000 companies ensure trust in the data that powers their business.