May 2026 Summaries

As data pipelines grow in complexity, data testing and data observability have become crucial concepts for maintaining data reliability. Data testing involves validating datasets against predefined expectations to ensure quality and stability, using automated checks for schema, freshness, volume, and business rules within data processing workflows. However, testing alone is insufficient as it only confirms known conditions; this is where data observability comes into play. Data observability continuously monitors data behavior across systems to detect unexpected changes, anomalies, and operational issues that may not have been predefined, thus providing a broader visibility into pipeline behavior and system health. Modern data teams increasingly rely on both methodologies, leveraging data contracts—a single, version-controlled YAML specification co-authored by engineers and business users—to tie them together. This integrated approach allows for precise validation of known data requirements while simultaneously catching unforeseen issues across the data ecosystem, ensuring a more reliable data infrastructure.

Data engineering involves the design and maintenance of systems that transport and transform data from its source into a reliable and actionable form for decision-making, analytics, and machine learning. The role of a data engineer has evolved beyond merely moving data to ensuring its accuracy, freshness, and usability throughout its lifecycle. This shift requires a focus on automation, observability, and governance, emphasizing the need for systems thinking over simple pipeline management. Data engineers must possess strong skills in SQL, Python, data modeling, and cloud infrastructure, as well as a keen understanding of distributed systems and data quality. The modern data stack is a layered system involving data ingestion, storage, transformation, and consumption, with orchestration and observability playing crucial roles. Best practices in data engineering include treating data as a product, implementing version control, automating validation, conducting early quality checks, and designing systems to handle failures visibly. As the field progresses, the emphasis on reliable data systems is becoming increasingly important, with data contracts, real-time processing, and built-in governance gaining prominence.

Data engineering is categorized into four distinct levels, each representing a progression in complexity and responsibility, with Level 1 focusing on SQL and basic ETL pipelines, Level 2 dealing with distributed systems and cloud data warehouses, and Level 3 encompassing streaming, orchestration, and big data architecture. Level 4, however, embodies a mature understanding that not every request necessitates building new pipelines, emphasizing the importance of restraint and data quality over complexity. This level challenges the prevalent industry focus on depth of stack and pipeline output, advocating instead for reliability and trustworthiness of data as the ultimate goals. The text highlights the over-engineering trap, where complexity is mistaken for capability, and stresses that reliable data, not extensive infrastructure, is the true deliverable. By focusing on data quality at every stage, engineers can build more effective systems, with Level 4 thinking encouraging a shift towards more efficient and trustworthy data solutions.

Financial institutions are transitioning from the high-level principles of BCBS 239 to the more concrete operational requirements outlined in the ECB's RDARR Guide, emphasizing the need for robust data quality controls across the entire data lifecycle. This shift demands that banks implement comprehensive data monitoring and governance mechanisms, transforming data quality into a strategic capability that involves business ownership, issue management, and executive-level reporting. Soda, a modern data quality platform, addresses these challenges by enabling automated data observability, auditability through data contracts, and integrated issue management, facilitating compliance with evolving regulatory expectations. Institutions face the "make versus buy" decision when integrating these capabilities, weighing the benefits of existing systems against the scalability and standardization offered by platform-based solutions like Soda. By effectively managing data quality, organizations can not only achieve regulatory compliance but also build a reliable data foundation for future initiatives.

Data observability is a practice that continuously monitors the health of data systems by focusing on five key pillars: freshness, volume, distribution, schema, and lineage. These pillars collectively provide insights into whether data is arriving as expected, maintaining its integrity, and behaving consistently, allowing teams to detect, diagnose, and resolve data issues before they impact decision-making processes. Unlike traditional data testing, which validates data against predefined rules, data observability identifies unexpected anomalies, offering a broader and faster coverage of potential data issues. Implementing data observability involves steps such as identifying critical assets, defining service level agreements, instrumenting signals, establishing baselines, and creating a triage workflow for efficient alert management and incident response. A robust data observability platform should offer stack-wide integrations, adaptive anomaly detection, and effective alert routing to ensure comprehensive monitoring and quick resolution of data issues, ultimately enhancing the reliability and trustworthiness of data pipelines and systems.