January 2022 Summaries

The article explores the evolving landscape of data management trends for 2022, highlighting the adoption of innovative tools and practices that are reshaping the industry. Key trends include a focus on data lineage and quality, which enhance reliability and productivity by documenting and storing data effectively to eliminate silos. The concept of data meshes is gaining traction, facilitating the sharing of knowledge and tools among data teams for better collaboration. There's a shift towards unified integrated development environments (IDEs) to streamline analytic work and reduce context switching. The decentralization of data responsibilities is emphasized as product teams take on more end-to-end data management tasks, necessitating tools that support scalability and resilience. Specialized tools are expected to evolve, enriching data as a product, while the proliferation of diverse data tools presents challenges and opportunities for integration. Data orchestration is becoming a crucial part of operational infrastructure, with a trend towards interconnected processes and the integration of data management with MLOps. The article underscores the importance of seamless integration between tools, adherence to open standards, and the growing focus on data visibility and governance to comply with regulatory requirements and make informed decisions.

Adding Data Quality to DAGs with Great Expectations involves using this tool in an Airflow DAG to perform data quality checks. This approach prioritizes what checks are done and how they're scheduled, allowing for iterative improvements to existing pipelines. By implementing unit tests and acceptance tests, data teams can start small and work towards covering all test cases. Great Expectations provides a Profiler that helps create Expectation Suites quickly, reducing boilerplate code and broadening the scope of data coverage. The tool enables checks on statistical analysis across tables or columns, allowing for more nuanced issues to be found in the data. By orchestrating data quality tasks within DAGs, teams can ensure critical pipelines are always scheduled with data quality checks. Great Expectations offers benefits such as compile-to-docs functionality, automated profilers, and an extensible library of built-in checks, making it a trusted tool for managing data quality lifecycle.