February 2024 Summaries

Conditional statements in Snowflake can significantly improve the efficiency of data sampling queries by filtering out irrelevant data and focusing on pertinent information. Three ways to enhance data sampling efficiency include using partitioned tables with sampling, incorporating `CASE` statements for dynamic adjustment of sampling rates or methods based on specific criteria within the data, and utilizing `JOIN` statements to sample data based on relationships between tables or within subsets of a single table. Each method has its own advantages and limitations, and the choice depends on the specific requirements and constraints of the analysis.

Metaplane is enhancing its Custom SQL monitors in February 2024 to improve their alignment with other monitor types and reduce volatility. The update will affect the expected ranges, which may shrink or expand. Anomaly state handling has been improved by freezing the expected bounds until the observed value returns within range, aligning it with other monitors. Additionally, "Mark Normal" now adjusts the model to maintain similar expected range sizes after anomalies are detected. Seasonality detection methods have also been standardized across all monitor types for better consistency.

Metaplane, a tool that helps teams understand downstream dependencies and stakeholders before making changes in their warehouse, has introduced a new feature allowing users to export lineage graphs as CSVs. The platform uses SQL parsing and metadata from the data stack to automatically generate column-level lineage. Users can now leverage this feature to trace upstream for root cause analysis of data quality incidents, determine impacted assets downstream, and proactively update stakeholders when making changes such as deprecating tables or schema alterations. The CSV export includes details like number of hops, types, URLs (if applicable), and Metaplane URLs. To export lineage, users can navigate to any lineage graph in their account and click the "Export" button. New users can sign up for a free account or book a workshop with data observability experts to get started with Metaplane.

Metaplane has updated its integration capabilities by adding support for Tableau Data Sources, allowing users to trace the root cause of issues more quickly. The update includes displaying Tableau workbooks, worksheets, and dashboards in column-level lineage graphs. By integrating Tableau with Metaplane, users can understand how data quality incidents impact their Tableau users and identify associated tables and fields for faster issue resolution. To connect Tableau to Metaplane, users need connection details and a user token, which can be found in the documentation. New users can sign up for free or book a workshop with data observability experts to get started.

Metaplane has announced an Airflow integration that provides observability into DAGs and tasks for long runtimes. The integration allows users to monitor the health of their data pipelines, identify root causes for data incidents using end-to-end lineage, and have a single pane of glass to view the entire data platform. Metaplane uses machine learning to automatically monitor and predict Airflow job durations, opening data incidents and sending alerts when jobs take longer than expected. The integration also helps identify root causes of data incidents using lineage.

Schema drift is inevitable in Snowflake, leading to missing or inconsistent data that compromises query and report accuracy. To track schema drift, users can generate and compare schema snapshots, use the Snowflake Community Python script (schemachange), leverage Snowflake's information_schema, or employ automated monitoring tools like Metaplane. By tracking schema changes, users can maintain data quality, integrity, and reliability while increasing trust in their organization's data systems.

Adaptive flatline alerts are designed to detect irregular patterns in data ingestion without causing false alarms. These alerts can be set up by modeling the data as a step function and establishing buffer time for potential delays or variances in insert timing. Metaplane's flatline detection algorithm learns about your data over time, providing more meaningful and actionable alerts tailored to your context.