September 2021 Summaries

Banks are under significant economic pressure due to the rise of FinTech companies, which have rapidly gained a substantial share of the personal loan market, and the necessity to adapt to digitalization. Despite technology offering opportunities like improved regulatory compliance and risk management, banks struggle to compete with less-regulated tech companies and face challenges in managing vast amounts of customer data effectively. The reliance on legacy systems and regulatory constraints hinder their ability to modernize infrastructure and fully utilize cloud computing. Apache Airflow® is presented as a potential solution, enabling banks to integrate legacy systems with modern data stacks without needing to migrate to the cloud, thus facilitating improved data orchestration, centralized data management, and enhanced AI deployment. Airflow's capabilities in automating and standardizing data processes are exemplified by its use in institutions like Societe Generale and ING Bexs Bank, helping them manage data workflows efficiently and overcome data silos, ultimately aiming to improve customer experiences and operational efficiency.

Apache Airflow® and Apache NiFi are two distinct open-source tools used to manage data, each catering to different needs within data management and workflow orchestration. Apache NiFi, originally developed by the NSA, excels in automating data flow between systems, making it ideal for handling large volumes of data through a user-friendly, drag-and-drop interface without requiring coding skills. It is particularly suited for long-running ETL processes and live batch streaming, despite its limitations in scaling and scheduling. In contrast, Apache Airflow® is a flexible task scheduler and data orchestrator favored for its robust capabilities in scheduling tasks, managing dependencies, and creating workflows as directed acyclic graphs (DAGs), written in Python. This makes it highly suitable for complex workflows and business-critical processes, supported by a vibrant community and a rich user interface. While Airflow is more widely adopted due to its versatility and active community, the choice between the two tools hinges on specific use cases: NiFi is optimal for basic big data ETL processes, whereas Airflow is the preferred option for orchestrating and executing sophisticated workflows.

Wise, a global fintech company known for its innovative money transfer solutions, leverages Apache Airflow to orchestrate its machine learning processes. Machine learning engineer Alexandra Abbas explains that Airflow is integral to Wise's data management, particularly in retraining models in Amazon SageMaker as part of their machine learning infrastructure. This includes transitioning to stream machine learning with tools like Apache Kafka and Apache Flink, which enhances scalability and speeds up production deployment. By decentralizing Airflow into team-specific environments, Wise ensures better scalability and security, allowing data scientists to collaborate and prototype efficiently. Airflow's user-friendly interface and versatility make it a preferred choice for Wise’s data orchestration needs.

The ETL (Extract, Transform, Load) process is a structured data management framework that involves extracting data from diverse sources, transforming it through integration and cleansing, and finally loading it into a data warehouse for efficient analysis. This process converts raw data into a cohesive format, enabling businesses to analyze trends and anticipate outcomes, thus supporting smarter decision-making. Key steps in ETL include copying raw data, establishing connectors, validating and filtering, transforming, storing, loading, and scheduling data processing. Companies can build ETL systems using batch or real-time processing, with tools like Apache Airflow enhancing pipeline management. While ETL transforms data before loading, the ELT (Extract, Load, Transform) approach, suitable for big data environments, performs transformations post-loading, each method offering different advantages depending on organizational needs. Adopting ETL processes, particularly with platforms like Airflow and Astronomer, can enhance business intelligence, resource management, and data-driven decision-making, offering significant returns on investment.