January 2023 Summaries

Astronomer has enhanced the integration of the Great Expectations tool with Apache Airflow by refining the GreatExpectationsOperator, making it more user-friendly and Airflow-centric. The latest version eliminates the need for a pre-defined Checkpoint, as it automatically creates a default one if not provided, and offers full customizability through the checkpoint_kwargs parameter. This update simplifies the process of running data quality checks by converting complex SQL and Python scripts into intuitive JSON templates called Expectations, applicable to various data formats, including pandas and Spark dataframes. The integration also supports dynamic task mapping, allowing for streamlined checks across multiple tables, and improves observability by seamlessly integrating with Astro's data lineage features. The enhanced operator maintains backward compatibility and simplifies the migration from other data quality tools by facilitating easy configuration and deployment, making it a compelling choice for organizations looking to enhance their data quality processes.

Astronomer has introduced a revamped workspace homepage for Astro, designed to enhance the daily experience of data practitioners by providing a consolidated view of essential information. The updated interface includes an operational overview that allows users to monitor the success or failure of DAG or task runs over customizable time frames, enabling quick identification of areas needing attention. The homepage also features onboarding resources and ongoing guidance through tutorials, documentation, and curated content, such as Providers, Modules, and DAG templates. Additionally, a "Jump back in" section offers shortcuts to recent activities and common tasks, facilitating seamless continuity of work. The new interface aims to boost productivity and streamline the Astro user experience, with demos available for prospective customers.