May 2020 Summaries

Pulumi has been recognized as a 2020 Gartner Cool Vendor for Agile and DevOps, highlighting its innovative approach to Infrastructure as Code (IaC) by leveraging common programming languages like JavaScript, Python, and Go to deploy infrastructure across major cloud providers such as AWS, Azure, and GCP. This recognition underscores Pulumi's impact on improving collaboration between developers and operators, allowing them to use industry best practices like unit testing, versioning, and error correction in IDEs. Pulumi's open-source platform, which is also available as a SaaS, supports policy as code and integrates with CI/CD pipelines to enhance compliance, cost control, and security. By providing a consistent programming model for various cloud resources, Pulumi has facilitated immediate organizational impact, enabling faster, confident deployment with standardized best practices. With features like infrastructure unit tests, reusable components, and policy enforcement, Pulumi has become a tool of choice for many organizations worldwide, promoting a modern, collaborative future in IaC.

Dustin Farris, a cloud engineering consultant, is leading a project to build a new data lake for a public university using Pulumi, focusing on handling sensitive student and research data while meeting strict quality and security standards. The project leverages modern cloud services, such as AWS Glue for ETL processes, AWS Lambda Step Functions for workflows, and AWS Lake Formation for secure data cataloging, to ensure superior performance, scalability, and security. Pulumi, favored for its support of TypeScript and modern IDEs, is used for Infrastructure as Code, enabling efficient testing and verification through resource mocking, which significantly accelerates the testing process from 20 minutes to 20 seconds. The team employs unit tests to prevent configuration errors, and integration tests to validate the entire stack, with plans to advance QA and security through Continuous Integration and Delivery using AWS CodePipeline and Pulumi's policy-as-code framework, CrossGuard.

Writing infrastructure policy using high-level programming languages, known as Policy as Code, enables the automation and enforcement of best practices, allowing organizations to apply software development methodologies like testing, automated deployment, and version control. Traditional cloud provider GUIs often lack repeatability and versioning, while policies written as code allow for testing against live or ephemeral systems, leading to significant organizational benefits such as cost savings, improved compliance, and efficient deployments. For cost control, policies can be created based on pricing data to limit spending and utilize cloud provider resources to manage unused resources. Compliance is enhanced by enforcing infrastructure policies that prevent unauthorized access and secure the software supply chain. Pulumi’s deployment engine, for instance, allows for policy validation before resource deployment, blocking non-compliant resources and supporting testing at various levels. Policy Packs enable bundling of related policies for organized management, while the integration of cloud-native tools like IAM Access Analyzer with Policy as Code ensures comprehensive management and protection of infrastructure. Ultimately, Policy as Code not only benefits developers and operators but also plays a crucial role in an organization's overall success by automating cost control, ensuring compliance, validating infrastructure, and encoding best practices.

Pulumi leverages general-purpose programming languages like C# and F# to provision cloud resources, enabling developers to utilize native tools and perform automated infrastructure tests, such as mock-based unit testing. This blog post delves into unit testing Pulumi programs, replacing remote calls with mocks to quickly and deterministically test infrastructure code without cloud dependency. Developers can construct a program to deploy a static website to Azure, using Test-Driven Development (TDD) to iteratively add tests and components. The process involves defining resources like a resource group, storage account, and website files, and verifying properties through unit tests. Mocks are implemented to emulate cloud behavior, addressing challenges such as non-deterministic remote calls and unsupported features like FileAsset conversion. The blog concludes with the successful testing of a static website deployment, emphasizing the speed and flexibility of Pulumi's testing capabilities for infrastructure code.

Over the past decade, Go has emerged as a prominent language for cloud applications, and with Pulumi 2.0, it extends its capabilities to manage modern cloud infrastructure. Pulumi is a versatile infrastructure as code tool that supports over 50 clouds, including AWS, Azure, and GCP, and is designed to accommodate a variety of workloads like containers, serverless, and Kubernetes. Built on Go, Pulumi offers the advantages of rich programming languages alongside desired-state infrastructure management, enabling consistent multi-cloud workflows. It caters to diverse cloud engineering teams, facilitating collaboration among developers, infrastructure, operations, and security engineers. Pulumi enhances Go's software engineering strengths, allowing users to deploy serverless functions, manage containers, and interface with Kubernetes seamlessly across different cloud environments. By leveraging familiar programming paradigms, Pulumi simplifies complex deployments and encourages the creation of strongly-typed, encapsulated APIs for cloud services, thereby empowering users to efficiently orchestrate their cloud resources.

In this final part of a series on securely managing AWS credentials within CI/CD workflows, the focus is on using Pulumi to update AWS resources securely while storing sensitive data. The article emphasizes the transition from using IAM User access keys, which have long-term drawbacks, to assuming IAM Roles that provide short-lived and task-specific permissions, thereby enhancing security. It details the creation of an IAM Role, WebsiteStackUpdaterRole, which is tailored to grant only the necessary permissions for specific Pulumi stack updates, ensuring that access is restricted to authorized users. The text also discusses how Pulumi securely stores sensitive data, highlighting the use of encrypted configuration and checkpoint files, as well as the option for users to implement custom secret providers for heightened security. Overall, the series provides a comprehensive guide to best practices in managing AWS access and safeguarding sensitive data in cloud environments using Pulumi.

The final installment of the Architecture as Code series delves into Kubernetes deployment patterns, exploring the nuances between single large clusters and multiple small clusters, each offering distinct advantages and challenges concerning resource efficiency, cost, multi-tenancy, and isolation. It highlights the flexibility of Kubernetes in supporting various deployment models and discusses multi-cloud strategies for maintaining high availability by distributing applications across different cloud service providers. The article illustrates the use of Pulumi, a tool that facilitates infrastructure as code by enabling reusable infrastructure components and supporting Kubernetes deployments on platforms like AWS, Azure, and Google Cloud. Additionally, it emphasizes the importance of architecture as code in embodying DevOps principles, promoting consistency, security, compliance, and cost management in modern application deployments.

Pulumi has enhanced its secret management capabilities to support "Cloud Secret Providers," allowing users to maintain exclusive access to their sensitive data when provisioning infrastructure. Initially, Pulumi offered secret encryption via passphrase or its service backend, but these options lacked the control users desired. Now, Pulumi integrates with several encryption services, including AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault. The article provides a detailed example of utilizing AWS KMS to encrypt secret values in a Pulumi stack, including setting up a KMS key, configuring IAM roles, and ensuring secure access through key policies. It demonstrates how to verify encryption by attempting Pulumi operations without access to the KMS key and explains how to manage Pulumi state files to ensure data remains encrypted. This approach not only strengthens security but also aims to facilitate compliance audits by ensuring that sensitive values are securely encrypted and accessible only to authorized users.

CrossGuard, Pulumi’s Policy as Code solution, has reached general availability as part of the 2.0 release, offering enhancements like improved CLI support for managing Policy Packs and the ability for organization administrators to enable or disable them using simple commands. The updated version provides policies with more resource information, such as options, parent, and dependencies, allowing for more comprehensive policy creation, like ensuring DynamoDB tables have scaling policies. Python support has been added, enabling the authoring of Policy Packs using the Python SDK, which can enforce policies across stacks written in any language. Configurable Policy Packs introduce flexibility, allowing organizations to adjust policies for diverse environments, such as restricting EC2 instance types differently in production and non-production stacks. These advancements aim to enhance the user experience for both enterprise and open-source users, with resources like webinars available for further learning.