June 2023 Summaries

Sematic has released a new update featuring significant enhancements, including GitHub integration, real-time metrics, and native dependency packaging. The GitHub integration allows Sematic Enterprise users to trigger test pipelines from GitHub pull requests and block PRs until the successful completion of these pipelines, providing a seamless connection between repositories and pipeline runs. Real-time metrics now enable users to visualize data during function execution directly within Sematic, bypassing the need for third-party tools like Weights & Biases or Tensorboard, although these remain options for more advanced visualization. Additionally, a new example for language modeling demonstrates the tool's flexibility, with a feature allowing visualization of language model prompts and responses. Sematic now supports native dependency packaging with Docker for Cloud Execution, simplifying the process of building and deploying container images without manual rebuilding. Users are encouraged to upgrade to the latest version for these improvements and to seek assistance via Sematic's Discord server if needed.

Apache Airflow, a widely adopted open-source project initially developed at Airbnb and now commercialized by Astronomer.io, is celebrated for its versatility in orchestrating data workflows with Direct Acyclic Graphs (DAGs) and its broad use in automating ETL processes. However, it faces significant challenges when applied to Machine Learning (ML) workflows, which require highly iterative development, local execution, comprehensive lineage tracking, and detailed visualizations—features not inherently supported by Airflow. ML workflows involve constant retraining, testing, and fine-tuning of models, necessitating a fast feedback loop and granular tracking of all contributing assets, which Airflow's architecture doesn't easily accommodate. While Airflow's extensive community support, stability, and multi-language capability are undeniable strengths, the platform's lack of advanced features for ML tasks has led many teams to seek additional tools or layers to bridge these gaps. As a result, despite its popularity, Airflow is often seen as inadequate for ML needs without substantial modifications.