November 2023 Summaries

Metaplane's latest Incident Page features allow users to efficiently navigate and prioritize fixes for specific incidents. With usage context, filters, and search capabilities, users can quickly identify and resolve data quality issues related to revenue data. The incident page also provides top-level stats on open incidents and highlights important objects with icons, making it easier to manage and triage ongoing or acknowledged incidents.

Metaplane introduces "alert excludes," allowing users to customize their alert routing rules by excluding certain tags or parts of the data stack from receiving alerts. This feature aims to reduce noise and ensure that users only receive alerts about the most important matters. The alert settings UI has been refreshed, making it easier to add rules and see which rules apply to specific alert destinations. Metaplane will be encouraging users to transition their custom SQL monitor rules into tag rules in the coming weeks.

The text discusses the forecasting of data engineering trends for 2024, based on the trending topics from 2023. These include Data Contracts, Data Mesh, Data Fabric, and Semantic Layer/Metric Layers. It also mentions how these trends are influenced by factors such as region, industry, company size, data strategy maturity, and social media followings. The author encourages readers to reach out with any disagreements or inaccuracies they may find.

Snowflake currently offers several table types including Base/Permanent, Temporary, and External. In addition to these, four new table types were announced between 2022-2023: Dynamic, Event, Hybrid, and Iceberg tables. These new table types represent the evolution of Snowflake's table types from duration-based and location-based to use-case based. Dynamic Tables allow users to materialize query results with an input for desired lag, creating a new table that references and merges data from upstream objects on a specified schedule. They are useful for tasks like cleaning and merging data from multiple sources in recurring modeling processes. Event Tables capture information about Snowpark or stored procedure job runs, providing records generally split into severity levels (e.g., WARN, ERROR) for debugging purposes. They are beneficial for anyone developing on top of Snowflake. Hybrid Tables support both high-volume transaction workloads and single-row lookups, making them suitable for companies looking to get rid of their transactional system and use Snowflake for analytics. Iceberg Tables, named after the table format created by Netflix and now Apache open-source licensed, are useful for handling large datasets and tracking schema evolution in metadata. The decision to leverage this new table type will likely be a broader organizational discussion about how data is used and stored.

The latest update allows users to sort children objects within warehouse data, including alphabetically. Previously, the list was sorted by the number of active incidents on the child object, which made it difficult to locate specific items. Now, users can also sort by the number of queries, downstream entities, and alphabetically. Additionally, minor UI improvements have been implemented for better readability.

The text discusses an update to monitor creation flow in a system. It explains how users can configure their monitors according to their teams' and business needs, including setting thresholds for receiving alerts on data quality issues, scheduling monitoring run frequency, and narrowing the monitored data with time windows or WHERE clauses. The text also mentions that certain configurations may differ between monitor types, such as custom SQL monitoring where users can choose machine learning model types and expected value formats of returned data. Additionally, it suggests that Metaplane will recommend monitors for immediate implementation when users click into their database, data warehouse, or data lakehouse integrations via the UI.

Metaplane has introduced a new list view feature that complements its existing lineage map. The platform now allows users to see how data flows through their stack, from Fivetran to the BI tool, at both the column and table levels. While the lineage map is useful for tracing data flows and showing relationships, sometimes a simple list view is more effective when looking for specific upstream or downstream lineages of tables, columns, or dashboards. The new list view displays direct and indirect lineages, broken down by the number of "hops" from the selected object.

Metaplane has introduced a new API endpoint called Evaluation History that allows users to access the evaluation history of their data quality monitors. This feature is beneficial for teams who prefer working in VS Code and rely on dashboards for daily standups, as it provides additional context on monitors' performance, including when they last ran, expected bounds, and whether they passed or failed.