The Origins, Purpose, and Practice of Data Observability

The concept of data observability, which refers to the degree of visibility into our data systems, is a modern adaptation of software observability that draws from control theory. Data observability helps maintain order within our data by preventing lapses in data quality and unacceptable data downtime. It goes beyond data quality monitoring by striving to understand why problems occurred and how they can be prevented. The four pillars of data observability are: metrics, metadata, lineage, and logs. These components are necessary and sufficient to describe the state of a data system at any point in time, in arbitrary detail.