December 2023 Summaries

Taktile, a low-code platform used by global credit and risk teams to design, build, and evaluate automated decision flows at scale, processes millions of daily tracking events, which need to be anonymized and loaded into its Snowflake Data Warehouse. To achieve this, Taktile chose dlt (data loading tool) as the instrumentation engine, running it on AWS Lambda due to its secure, scalable, and lightweight nature. dlt's ability to automatically schema detect, normalize unstructured data, and provision staging destinations makes it a powerful tool for event ingestion. By using AWS SAM, a lightweight Infrastructure-As-Code framework, Taktile can easily deploy and manage their resources. However, there are caveats to be aware of when setting up dlt on AWS Lambda, such as handling local files and database secrets. Despite these considerations, dlt and AWS Lambda form a great foundation for building a production-grade instrumentation engine.

A student created an AI-enhanced email analysis tool using Kestra and dlt to automate the process of extracting data from Gmail, analyzing sentiment with OpenAI, and notifying results via Slack. The tool loads data into BigQuery and provides insights, reducing manual intervention and errors. Kestra is an open-source automation tool that makes workflows easy to create and manage, while dlt is a library for loading data from various sources into structured datasets. The student set up two flows in Kestra, one extracting email data and the other analyzing sentiment, and used a schedule trigger to execute the flow on a regular basis. The outcome is a Slack assistant delivering crisp inbox insights and a well-organized table in BigQuery ready for further analysis.