Company
Date Published
Author
Dustin Farris
Word count
1553
Language
English
Hacker News points
None

Summary

Dustin Farris, a cloud engineering consultant, is leading a project to build a new data lake for a public university using Pulumi, focusing on handling sensitive student and research data while meeting strict quality and security standards. The project leverages modern cloud services, such as AWS Glue for ETL processes, AWS Lambda Step Functions for workflows, and AWS Lake Formation for secure data cataloging, to ensure superior performance, scalability, and security. Pulumi, favored for its support of TypeScript and modern IDEs, is used for Infrastructure as Code, enabling efficient testing and verification through resource mocking, which significantly accelerates the testing process from 20 minutes to 20 seconds. The team employs unit tests to prevent configuration errors, and integration tests to validate the entire stack, with plans to advance QA and security through Continuous Integration and Delivery using AWS CodePipeline and Pulumi's policy-as-code framework, CrossGuard.