October 2022 Summaries

Five insightful talks from Coalesce 2022 focused on career development in the data industry. Ian Fahey's talk used roleplaying games to teach about identifying one's strengths and weaknesses, while Stephen Bailey discussed being an effective generalist. Brittany Krauth highlighted the transition from analyst to analytics engineer, Michael Chow shared his experience as an accidental analytics engineer, and Liz Bianco-Lane and Meghan Cassidy provided a career progression framework for individual growth. These talks emphasized reflection, understanding one's strengths, and adapting to new roles in the data industry.

Metaplane has partnered with Snowflake to enhance trust in data by providing a comprehensive solution for data observability. The partnership aims to empower data teams of all sizes with superior data quality, aligning perfectly with Snowflake customer needs. By leveraging Snowflake's advanced technical foundation and Metaplane's expertise in data observability, the collaboration will enable users to power critical use cases with confidence, alerting them to data anomalies and providing context for diagnosing downstream impact and analyzing upstream root cause.

Metaplane has integrated with GitHub to provide impact analysis on dbt pull requests (PRs). This integration allows data teams to know which downstream models and important business intelligence assets will be affected by changes made to dbt models. The tool can also proactively inform users about potential root causes for data incidents using dbt PR history. By parsing column-level lineage in the warehouse and BI tools, Metaplane analyzes changes made to dbt models and posts comments providing downstream impact analysis before merging the PR. Additionally, an impact analysis report is available within the Metaplane web app for sharing with stakeholders.

Horizon Data is a South African-based data consultancy that helps high growth startups scale their data platforms. Founded in August 2021 by Rijnhardt Kotze, the company provides services such as data architecture, embedded data teams, and maintenance after project completion. One of the biggest challenges faced by Horizon Data was getting ahead of schema changes and data quality issues. To address this, they adopted Metaplane, a tool that enables proactive alerts for data quality issues. The adoption of Metaplane has helped Horizon Data transition from reactive to proactive monitoring, improving efficiency and trust in the data.

Data quality is crucial for the success of an organization as it directly impacts business performance. Poor data quality can lead to incorrect decisions, increased costs, and reduced revenue. Key dimensions of data quality include intrinsic dimensions such as integrity, accuracy, completeness, consistency, freshness, and privacy/security, and extrinsic dimensions like relevance, reliability, timeliness, usability, and validity. Common challenges in maintaining high data quality standards include software sprawl, data proliferation, lack of metadata, typos and other human errors, and difficulty hiring experienced team members. To measure and improve data quality, organizations should identify what matters, pinpoint pain points, make metrics actionable, and consistently monitor their data quality. A People, Process, Technology (PPT) framework can be applied to improve data quality by involving teammates in metric development, implementing business processes for improvement, and utilizing technologies like data observability tools and automation solutions.

Incidents is a new feature by Metaplane that groups related failing tests and presents interactive summaries with potential root causes, aggregate downstream impact, usage, and more. It helps data practitioners identify and address data quality issues faster by synthesizing failures into actionable stories called "data incidents." By grouping together related data quality issues based on issue type, lineage, and time correlation, Metaplane can send highly actionable alerts and reminders, provide a central place to view potential root causes and downstream impacts, and combat alert fatigue. The feature is now available to all customers, including the free forever plan.

Metaplane introduces enhanced control over tested data by allowing timestamp column filters. This feature enables users to configure tests with rolling time windows, monitoring only a subset of data within a specified time frame. It is particularly beneficial for large or incremental datasets where recent data holds significant importance.