January 2020 Summaries

ACME E-Commerce, a growing online retailer, embarked on an analytics journey to optimize its ad performance using AWS technologies. Initially, they used Amazon Redshift for analytics due to its ease of setup and scalability for BI workloads. However, as their ad platform scaled, the limitations of Redshift became apparent, such as resource-intensive maintenance and operational issues under heavy loads. To address these challenges, ACME transitioned to a serverless architecture using AWS services like Kinesis, Firehose, Glue, S3, and Athena, enabling them to handle large-scale data with low operator intervention and reduced costs. They utilized Pulumi to implement this architecture as code, simplifying infrastructure management and fostering automated and reproducible environments. This shift not only enhanced scalability and cost-efficiency but also laid the groundwork for future analytics initiatives by encapsulating best practices into a reusable library. Despite the advantages, the new architecture posed challenges, including managing processing time versus event time and optimizing file sizes for query performance.

Pulumi's dynamic providers offer a versatile approach to manage infrastructure tasks when existing resource providers fall short. These providers enable users to implement custom logic during the CRUD life-cycle of resources through life-cycle callbacks, allowing for more tailored infrastructure management. While dynamic providers are currently supported only in JavaScript, TypeScript, and Python, they provide substantial flexibility by enabling interactions with cloud resources and external services via REST APIs. Examples include enabling Azure Storage's static website feature and adding custom domains to Azure CDN endpoints, tasks not directly supported by standard resource providers but achievable through dynamic providers. This functionality allows developers to execute scripts or manage configurations on virtual machines across different cloud providers, illustrating the adaptability of dynamic providers in addressing infrastructure provisioning challenges.

Navigating the complex landscape of AWS container services can be overwhelming, but deploying containers using infrastructure as code, particularly with Pulumi, offers a streamlined and reproducible approach. The process involves using the AWS Elastic Container Registry (ECR) for storing and managing container images, which provides a scalable and highly available solution. Developers can choose between AWS's Elastic Cloud Service (ECS) and Elastic Kubernetes Service (EKS) for container orchestration, with ECS offering seamless integration with AWS services and EKS providing portability and fine-grained control through Kubernetes. Depending on workload requirements, containers can be run using AWS Fargate for smaller, scalable tasks or EC2 for larger, resource-intensive applications. Pulumi's framework allows the declaration of repositories, creation of ECS or EKS clusters, and configuration of services like load balancers, facilitating the deployment and management of containerized applications. The use of Pulumi's Crosswalk for AWS simplifies these tasks by integrating frequent practices and providing a cohesive interface for deploying infrastructure on AWS, thereby making it easier to build, store, and manage AWS containers.