April 2023 Summaries

In 10 years, Data Observability is predicted to evolve significantly with advancements in technology and use cases. The core functional areas of incident detection, triage, and resolution will remain constant while the underlying technology and use cases change. Automation, genericity, and integration across workflow tools are expected to maximize monitor coverage and depth while minimizing setup time, cost, and management. Data Observability is predicted to become a "no-brainer" part of data stacks, with full visibility of all data flows from beginning to end through a Mission Control panel.

The latest update from the data team introduces a new feature, WHERE Clause for Monitors. This allows users to specify which segments of their data they want Metaplane to monitor. Business needs often change, requiring adjustments in models and dashboard filters. The WHERE clause can be used to focus on specific segments of data, such as changes in business or a particular area of the business. Users with existing accounts can apply this feature directly to new or existing monitors, while those without an account are encouraged to start with Metaplane's free plan or contact them for best practices implementation and usage.

Data observability is a proactive approach that focuses on ensuring high-quality data by providing visibility and insights into data pipelines and infrastructure. It addresses questions such as whether the data is correct, how up to date a dashboard is, or why a report broke. Adopting data observability practices offers several benefits to both businesses and data teams, including better-informed decisions, optimized processes, increased efficiency, and reduced troubleshooting time.

Metaplane introduces "Monitors as Code" for dbt users, allowing them to manage all configuration for their models in one place. This approach provides benefits such as a historical record of the configuration and an option for teammates to review and approve before shipping. Automatic creation, update, and disabling of monitors are possible through Metaplane's YAML-based monitor configuration. Manual syncing is also available in dbt. Monitors created through dbt can be viewed in Metaplane's UI, ensuring no unwanted changes or deletions from the Metaplane UI. This method offers fully automated monitoring management that fits seamlessly into a team's engineering workflow.

Metaplane has redesigned its incidents experience to provide more context, better root cause analysis, and improved communication for users dealing with data anomalies. The new Overview tab collects important information about the incident, while the Root Cause tab displays related pull requests, queries, similar incidents, upstream lineage, and a timeline of the incident. Additionally, Metaplane's Acknowledge feature allows users to acknowledge an incident for a set period of time, muting future notifications. These improvements aim to make resolving data issues more efficient and effective.

Metaplane now supports integration with Databricks, allowing users to continuously test their data for anomalies, monitor query performance over time, and generate upstream dependencies of BI assets. This integration is particularly useful for customers who use Databricks lakehouse, a platform that combines the best features of data warehouses and data lakes. Metaplane's support for Databricks offers an efficient solution for ensuring trusted data within teams.

The text discusses a new feature called "Metaplane Daily Updates" that aims to help data engineers manage their workload more effectively by consolidating less-timely updates into one convenient spot. This includes sync issues, offline monitors, and unresolved incidents. The update also provides valuable insights such as identifying which monitors are offline and when they stopped running. Users can now link directly from the Slack alert to a view of all offline monitors for a given database. Timely updates about new incidents and schema changes will still be received. This feature is available for all Metaplane customers who use Slack.