April 2020 Summaries

Scott Lowe, a veteran IT professional and Staff Kubernetes Architect at VMware, shares his experience using Pulumi to manage and test AWS environments for Kubernetes projects, particularly the Cluster API project. Cluster API is a Kubernetes initiative that employs a declarative API to manage Kubernetes cluster lifecycles, allowing for seamless management of clusters both on-premises and in the cloud. Scott uses Pulumi, an infrastructure-as-code tool, to quickly define, test, and deploy infrastructure across multiple AWS regions, facilitating the testing of Kubernetes and other related technologies. By employing Pulumi projects written in TypeScript, he manages consistent configurations across regions, creating a versatile test environment that supports the spinning up of VPCs, security groups, and instances necessary for Kubernetes clusters. This infrastructure-as-code approach has significantly streamlined his process, enabling rapid deployment and teardown of environments, which would otherwise be time-consuming without such tools.

Serverless architecture is a popular cloud computing model that allows developers to build and deploy applications without managing server infrastructure, offering advantages such as cost efficiency and scalability. This approach supports various programming languages and can be implemented using cloud services like AWS Lambda, Azure Functions, and Google Cloud Functions, which bill users only when their functions are executed. Serverless applications can be synchronous or asynchronous and often utilize REST interfaces to handle HTTP operations. Despite its benefits, developers still need to manage ancillary services like clusters, security groups, and load balancers, which can be efficiently handled using Infrastructure as Code (IaC) tools like Pulumi. Additionally, strategies such as warming, provisioned concurrency, and autoscaling can mitigate latency issues known as "cold starts." Overall, serverless architecture offers flexibility and efficiency, though it requires careful planning and resource management to optimize application performance and infrastructure deployment.

Pulumi 2.0 marks a significant milestone in the development of the open-source infrastructure as code platform, enhancing its ability to support modern cloud architectures with new features termed as "superpowers." This release extends the platform’s offerings by achieving language parity across Node.js, Python, .NET, and Go, and expanding the supported cloud resource providers to over three dozen. Pulumi 2.0 introduces improved CI/CD integrations, environment management capabilities, and architecture support, enabling teams to build complex systems using abstraction and packaging mechanisms. The new policy framework, CrossGuard, allows users to define and apply policies in familiar languages to ensure compliance and prevent errors during deployment. Additionally, the update enhances testing capabilities through unit and integration testing, and introduces "The Pulumipus," a new mascot to engage the community. Pulumi 2.0 aims to empower cloud engineering teams by allowing developers, operations, and security engineers to collaborate effectively without silos, making it easier to adopt and innovate with modern cloud solutions.

Pulumi has made Go a first-class language for building cloud infrastructure, allowing developers to manage environments like AWS, Azure, and GCP, as well as Kubernetes infrastructure using Infrastructure as Code. With the release of Pulumi 2.0, users can seamlessly deploy and manage Kubernetes resources, such as Pods, with Go, benefiting from strongly-typed inputs and outputs and reusable components for cleaner infrastructure design. Pulumi enables a productive experience for Go developers by integrating Kubernetes deployments with cloud providers and allowing the embedding of Pulumi programs within Go-based tools. By leveraging tools like the pulumi-docker package, developers can deploy custom Docker images and integrate them into Kubernetes clusters, offering a streamlined process that automatically handles updates and versioning of application code alongside the infrastructure. Pulumi's approach combines infrastructure and application deployment in concise, strongly-typed Go code, exemplified through various use cases such as deploying GKE clusters in GCP and managing complex Kubernetes applications efficiently.

The Pulumi team faces challenges in managing cloud costs due to the necessity of accessing multiple cloud providers for product development and testing, which can lead to uncontrolled cost increases when developers fail to properly clean up resources. To address this, Pulumi leverages an AWS Lambda Go application to automatically remove unnecessary cloud resources daily, using tags to track resource ownership and prevent deletions. By employing Pulumi and AWS Lambda, the team can monitor and reduce costs effectively, as demonstrated through a detailed process of deploying a cloud cleanup solution targeting unattached EBS volumes. This solution involves creating a lambda function with specific roles and policies, deploying it across multiple AWS regions, and iterating on the application to achieve significant cost savings, reportedly reducing daily AWS expenses by 64%.

The third article in a series about Architecture as Code explores implementing microservices as reusable components using Pulumi. It explains that microservices architecture involves independent services representing specific capabilities, which can be deployed without service interruption and communicate via HTTP, typically through REST interfaces. The article discusses how Pulumi organizes projects and stacks, allowing each microservice to be deployed as an individual stack, enabling independent development and deployment with enforced security policies through Role-Based Access Control. Pulumi uses a StackReference resource to facilitate communication between microservices, allowing shared access to networking and configuration data. The article provides a practical example using AWS, illustrating how Pulumi's approach to Architecture as Code helps create reusable components, such as a networking stack with a VPC class and a database component from RDS. It encourages readers to experiment with microservices deployment using Pulumi stacks and offers resources for further learning.

AWS provides best practices for tagging resources to facilitate cost tracking, automation, and organization, but ensuring consistent implementation across all projects can be challenging. This article discusses utilizing Policy as Code to enforce tagging strategies and Infrastructure as Code techniques to automate the application of tags across resources, ensuring adherence to organizational policies. Tags are key/value labels that help manage, search, and filter AWS resources and can include technical, automation, business, and security tags based on business requirements. The article demonstrates how to apply and enforce tagging policies using Pulumi's policy packs and highlights an automated approach to inject tags into AWS resources using global stack transformations, eliminating manual errors and simplifying the tagging process. The approach not only ensures compliance with tagging requirements but also streamlines the management of AWS resources by automatically applying necessary tags, thus reducing manual effort and potential oversight.

Tasia, a Computer Science student at the University of Washington, shares her experiences as the first intern at Pulumi, a startup company. She chose Pulumi for its startup environment, knowledgeable engineering team, and a product she found compelling, along with competitive compensation and learning opportunities. During her internship, she worked across the company’s Service and Platform Teams, contributing to projects such as adding support for tags and Docker images, and updating code generation tools for Pulumi 2.0. Despite the challenges of remote work, Tasia learned valuable lessons, including the benefits of writing tests before code and the complexities of cloud infrastructure. Her time at Pulumi was unexpectedly organized and mature, contrary to her expectations of startup chaos, and she left with a strong sense of accomplishment and excitement for the future.

Pulumi has introduced a new resource-oriented documentation format to enhance user experience by consolidating each resource on a single page, improving navigation and discovery. This update supports multiple programming languages, allowing users to switch between languages like Python, Go, and .NET for code examples and access deep-linked API documentation for each language. The new format is designed as a canonical, language-agnostic reference across all Pulumi resources, initially rolled out for AWS, Azure, GCP, and other cloud providers such as Kubernetes. Pulumi encourages user feedback on the new documentation through designated channels on their site.