August 2023 Summaries

Transactional databases are crucial components in modern data stacks, often serving as the foundation for other tools and processes. Metaplane offers deep integrations with MySQL, SQL Server, and PostgreSQL to ensure that your data replication process is functioning correctly and provide a comprehensive view of your entire data stack's quality. Key points: 1. Transactional databases are essential in modern data stacks for handling various types of data. 2. Metaplane offers integrations with MySQL, SQL Server, and PostgreSQL to monitor data replication processes and maintain data quality. 3. Schema changes can impact downstream queries and dashboards; Metaplane's schema change feature alerts users about any modifications in the database structure. 4. Common database replication issues include connection interruptions, unexpected schema drift, and atomicity problems. Metaplane's monitors help detect these issues by identifying duplicate or missing data. 5. Understanding end-to-end lineage is crucial for tracing the root cause of issues and refactoring data; Metaplane simplifies this process by allowing users to trace the lineage of a column through various systems in the data stack.

Metaplane introduces user roles and permissions, allowing users to be designated as viewers, editors, or admins. This feature is designed for data-driven companies where multiple teams need access to the platform. Editors can create, edit, and disable monitors, while viewers can only view everything but cannot add or edit monitors or manage incidents. Roles and permissions are simple to set up in the "Users" section of Settings.

The text discusses various considerations for maintaining data quality when adding new tools to a data stack. It covers scenarios such as cloud warehouse migrations, business intelligence tool implementation, and data ingestion and transformation processes. Metaplane is introduced as a solution that helps monitor these areas of potential data quality issues.

Snowflake Dynamic Tables are a new type of table that update when their parent table(s) update, useful for modeling where you need the data to be current. dbt is a popular framework for transformations, increasing accessibility and collaboration in modeling through SQL-based modeling and centralized code repositories with documentation history. While Snowflake Dynamic Tables and dbt have similarities such as updating an object based on query results and using SQL as the primary modeling language, there are differences including frequency of updates, modularity, version control, validation rules (tests), and lineage + documentation. Using both tools together can provide the benefits of a git environment for modeling code, built-in lineage and documentation, and automatic updates to models triggered by refreshed data.

Metaplane has released an upgraded monitoring platform with several new updates including a redesigned monitor page, improved group by monitors, enhanced visibility for Snowflake customers, and accelerated development capabilities. The new monitor page features a graph displaying the historical state of the monitor, clearer values predicted by ML models, and an overview section showing high-level information about the monitor. Group by monitoring has been overhauled to allow users to segment data by distinct table segments and add multiple different segment monitors to a single table. Additionally, Metaplane now captures metadata for every table or view in Snowflake or BigQuery warehouses, allowing users to access historical information about their data. The platform's re-architecture enables faster development of new monitor types and capabilities.

Metaplane introduces its latest evolution in data observability platform, offering smart monitor suggestions and the ability to monitor everything from Snowflake spend to dbt jobs and more. The new monitoring platform aims to solve common problems with traditional data observability tools by making it easier to set up and use. Key features include: automatic metadata collection, suggested monitors based on usage and insights, group by monitors for logical table segments, source-to-target monitoring (coming soon), and dbt job monitoring. Metaplane's mission is to help every team trust their data, regardless of size or bandwidth available.