March 2023 Summaries

Metaplane introduces suggested monitors to help data engineers manage complex data environments more efficiently. The new feature provides personalized recommendations based on usage, lineage, column name and type, and insights from similar customers. Users can access these suggestions through badges on individual table and column pages or by navigating to the "Monitors" tab in their account. This feature aims to free up data engineers' time for more impactful work.

Metaplane introduces new monitor types, including SUM, Percent Negative, and Percent Zero, to enhance its monitoring capabilities for various business metrics. The SUM monitor helps track important metrics like revenue or product usage time. The Percent Negative monitor identifies negative values in specific columns, while the Percent Zero monitor ensures that certain column values are either zeroed out or not spiking unexpectedly. These monitors can be applied to any numeric column within Metaplane's platform.

Metaplane has introduced an improved monitor management experience for its users, featuring enhanced filtering, bulk actions, and increased visibility into the status of monitors. The new "Monitors" tab allows easy navigation through all monitors within a database, schema, or table, with options to filter by name, perform bulk actions such as resetting, updating, and disabling multiple monitors at once, view offline monitors, and access disabled monitors for potential re-enablement. This update is also available for dbt job monitors.

Column-level lineage is a crucial aspect in the data space, as it traces data not just from table to table but also from column to column. This provides valuable insights into how data moves and can enable finer-tuned root cause analysis and more robust downstream issue prevention. Metaplane's approach to column-level lineage involves building a complete SQL parser using the ANTLR library, which converts raw SQL statements into an abstract syntax tree (AST). The AST is then used to walk through the SQL statement and pull out relevant data and context into an intermediate representation (IR), allowing for the determination of column-level lineage relationships. Despite challenges such as dealing with complex SQL statements and nested CTEs, building their own parser has been key in enabling comprehensive column-level lineage support at Metaplane.

Metaplane introduces its new column-level lineage map, designed to help modern companies trace their data flow through complex stacks. The tool provides a holistic view of how data flows from tables and views to BI tools, making it easier for teams to prevent downstream issues, diagnose root causes, ensure heavily-referenced objects are monitored, and raise data awareness within the company. With intuitive controls and visualizations, users can easily explore their lineage from source to destination, improving efficiency and collaboration among data professionals.

Metaplane has integrated with dbt Core, allowing users to unlock real-time monitoring for their dbt jobs with model granularity. The integration also leverages table-level lineage information from dbt and displays model-to-model connections in Metaplane's rich lineage view. This is the first step in expanding Metaplane's monitoring and data quality capabilities for dbt Core customers, with more features to be added in the future. For more information on the integration, visit the documentation at https://docs.metaplane.dev/docs/dbt-core.

Metaplane introduces its first public API for data engineers, aiming to enhance the monitoring process by providing more control over when and how data is monitored. The API offers custom automation of monitor runs, automated setup for complex data changes, and context retrieval for assessing data health and behavior. Users can leverage this API in various ways, such as integrating it with a GitHub action or pulling monitoring statuses into BI tools. Metaplane's team is available to assist users in understanding the new API and looks forward to seeing its applications.