November 2022 Summaries

The extensive amount of global data, estimated to be hundreds of zettabytes, presents a significant challenge and opportunity for data analysis, necessitating the use of specialized databases like data warehouses for meaningful insights. This text introduces Amazon Redshift as a powerful data warehouse solution, detailing the process of setting up a single-node Redshift cluster using Pulumi within an Amazon VPC and loading sample data from Amazon S3. The guide walks through configuring the necessary AWS infrastructure, including VPC, subnets, IAM roles, and S3 bucket, to ensure secure and efficient data processing. It concludes with a brief demonstration of importing data into Redshift and hints at future enhancements involving ETL pipelines and automation using AWS Glue, illustrating the potential for more sophisticated data management and analysis in subsequent steps.

Crosswalk for AWS, introduced by Pulumi, provides a library of components designed to streamline AWS infrastructure setup and adoption of best practices, now available in all Pulumi languages. The recent 1.0 release of its core libraries—AWSX, EKS, and AWS API Gateway—offers a stable and supported foundation to build and manage AWS-based applications, with thousands of Pulumi customers already utilizing these components. Pulumi's Crosswalk for AWS aids in configuring VPC networks, deploying containers on ECS or Fargate, setting up production-ready EKS clusters, and creating serverless architectures with AWS Lambda and API Gateway, all with minimal code. The upgrade to version 1.0 includes some breaking changes for TypeScript users, but existing functionality can be preserved by using the classic namespace. Pulumi continues to support AWS as a primary platform, offering tools for scalable and complex cloud applications. Crosswalk for AWS remains free and open source, encouraging users to start building their AWS infrastructure efficiently.

AWS Lambda has evolved significantly as a serverless platform, introducing new capabilities and performance enhancements that reduce the need for deploying full virtual machines or clusters. One notable improvement is the introduction of AWS Lambda SnapStart, which addresses cold start latency issues, particularly in Java-based functions, by using a snapshot of the function's fully-initialized execution environment. This results in up to a 10x improvement in startup performance without additional costs or significant code changes. SnapStart utilizes Firecracker MicroVM snapshots to store the memory and disk state, allowing for rapid environment resumption. However, developers must consider specific requirements, such as ensuring unique random numbers and managing network connections and ephemeral data. The Pulumi team demonstrated that SnapStart significantly reduces cold start times, making Lambda functions more responsive and expanding their applicability. Overall, these advancements make serverless computing via AWS Lambda a more compelling option for modernizing infrastructure while maintaining responsiveness.

Pulumi has introduced an experimental feature called Update Plans to provide users with a strong guarantee about changes made to their infrastructure, especially in critical and production environments. This feature, available in the Pulumi CLI version 3.48.0, allows users to save results from a pulumi preview to a plan file, ensuring that subsequent updates adhere strictly to the actions specified in the plan. Initially requiring the use of specific CLI commands and an experimental environment variable, Update Plans are now more accessible through a new option in the interactive update confirmation prompt, which aligns with typical user habits. Users can now choose to execute an update using Update Plans during interactive sessions, and any discrepancies between the preview and the update will be reported. Although in public preview and recommended for non-critical scenarios, Pulumi encourages users to try the feature and provide feedback through their community channels.

Self-service infrastructure is a key component of DevOps that enables application teams to provision their own infrastructure, enhancing the speed and efficiency with which IT organizations can achieve their goals. This concept is supported by AWS Proton, a service that allows platform engineers to create self-service infrastructure templates for deploying cloud resources in an automated manner. Recently, AWS announced integration with AWS CodeBuild as a provisioner in AWS Proton, which allows platform teams to define custom workflows using their choice of Infrastructure as Code (IaC) tools, such as Pulumi. CodeBuild facilitates the creation and management of Docker build environments, and its integration with Proton enables the execution of arbitrary commands for infrastructure provisioning and deprovisioning. This process involves creating a manifest file that outlines the steps needed for infrastructure management, and a schema file for self-service inputs, which informs application teams about the configurable parameters of the template. Templates can be stored in git repositories for version control, and Proton offers a user interface for application teams to deploy infrastructure using these templates, with the ability to view build logs and outputs. This integration provides the benefits of self-service infrastructure while leveraging Pulumi's capabilities for secrets management and stack history.

Pulumi has recently introduced several updates and enhancements across its platform, focusing on expanding its infrastructure as code capabilities and improving user experience. Key developments include the announcement of Pulumi Deployments, which allows infrastructure deployments as a managed service through various methods such as click-to-deploy, Git push-to-deploy, and programmatic deployments via REST API. Pulumi YAML has reached General Availability, providing a simplified declarative interface for infrastructure management. Additionally, the company added architecture-specific templates for common infrastructure use cases and introduced project-level configuration support to help manage growing complexity. Significant updates have been made to Pulumi's cloud providers, including new resources for AWS, Google, and Azure, along with enhanced Kubernetes support through updates to the Pulumi Kubernetes Operator and the introduction of a new provider for Flux. The Pulumi CLI now features elapsed time display for resource modifications, and the Automation API supports parallel execution of NodeJS programs. Other improvements include Python SDK updates, TypeScript version bumps, and token authentication in Go providers. Overall, these enhancements aim to streamline infrastructure management and support scaling needs for organizations using Pulumi.

Denis Willett, a software engineer at the North Carolina Institute of Climate Studies, works on the NOAA Open Data Dissemination (NODD) Program, which provides free and publicly accessible environmental data across major cloud platforms. NODD supports over 220 datasets and more than 24PB of open data, emphasizing quick and resilient data delivery. To manage the complex multi-cloud infrastructure, the team transitioned from using individual cloud provider tools to employing Pulumi, an infrastructure as code (IAC) platform. This shift allowed the team to automate processes, reduce manual errors, prevent infrastructure drift, and enhance collaboration by using a common programming language, Python. Pulumi's ability to facilitate multi-cloud management with programming control structures streamlined the deployment of infrastructure, improving the efficiency and scalability of managing NOAA's data resources.

Pulumi's Architecture Templates streamline the process of deploying serverless applications to cloud providers like AWS, GCP, and Azure by providing pre-configured starter templates in various programming languages. These templates include all necessary permissions and a sample function, allowing developers to focus on building their applications without the need to manage underlying infrastructure. By executing a simple CLI command, users can generate the necessary boilerplate code and deploy their application with Pulumi's CLI connected to their preferred cloud provider. The templates also include a basic website setup that displays the current date and time using a serverless function, ready for customization. Pulumi's approach ensures that best practices are followed from the start, simplifying cloud deployment and allowing developers to concentrate on their application logic. Pulumi continues to expand its offerings with templates for different applications and use cases, making cloud engineering more accessible and efficient.

Pulumi has introduced a CLI usability enhancement that displays the time taken to deploy each resource, offering users a clearer understanding of deployment durations across different operations such as up, destroy, import, and refresh. This feature alleviates the uncertainty of deployment times by providing concrete numbers, allowing users to compare the efficiency of deploying resources on different cloud providers like AWS and GCP. While the feature helps quantify deployment durations, it acknowledges potential errors due to factors like cloud provider load, availability zones, and resource readiness. Feedback from users is encouraged to further refine the CLI experience, with suggestions welcome on GitHub.

Pulumi has introduced project-level configuration, allowing users to set default configuration values for all stacks in a project through the Pulumi.yaml file, a popular request among users. This feature enables consistent settings across stacks, such as selecting a default AWS region or disabling default providers, while allowing overrides for specific stacks via Pulumi.<stack>.yaml or the CLI. Additionally, Pulumi now supports defining configuration schemas, aiding in the validation of settings through a structure similar to JSON schema. The recent developments also include the deprecation of the previous configuration key in favor of the new project-level setup, with plans for further improvements as outlined in a GitHub comment. Pulumi encourages users to provide feedback through its Community Slack and GitHub Discussions.

Pulumi YAML 1.0 has achieved General Availability, offering a stable, simple, declarative interface for Pulumi's Universal Infrastructure as Code platform, which is ideal for smaller-scale projects and serves as an entry point into Pulumi's ecosystem. This release includes key features such as full support for the pulumi convert feature, allowing seamless transition of YAML programs into other Pulumi-supported languages like Python, Java, Go, TypeScript, and C#, thus aiding in scaling up infrastructure complexity. Additionally, it provides improved IDE support with extensions for VS Code and Emacs, enhanced type checking and diagnostics, and a new str package for string manipulation. Pulumi YAML also enables the Pulumi Kubernetes Operator to define resources directly within Kubernetes YAML, offering a streamlined approach to manage cloud resources. This launch aims to cater to cloud teams familiar with markup languages by providing an on-ramp to more complex infrastructure management while retaining the ability to evolve with organizational needs.

Pulumi Deployments is a new suite of features designed to streamline and automate cloud infrastructure management, helping organizations scale their cloud footprint effectively without increasing headcount. It introduces tools like the Deployments REST API, Deployment Visualization, and both Click to Deploy and git push to Deploy functionalities, which offer simplified, code-free infrastructure deployment and management. These features are built on top of the Automation API, allowing for remote orchestration of cloud resources and providing scalability, observability, and security. The platform supports drift detection, temporary infrastructure management, and multi-region deployments, promising to turn the cloud from a bottleneck into a business accelerant. Currently in preview, Pulumi Deployments is available with free access and plans for usage-based pricing in the future.

Pulumi has announced significant advancements in its platform to enhance the scalability of cloud infrastructure through innovative features such as Pulumi Deployments, Pulumi YAML General Availability, Architecture Templates, and Project-Level Configuration. Pulumi Deployments offers a managed service for executing infrastructure deployments, allowing users to automate processes through various methods including GitHub integrations and a REST API, thereby improving scalability and security. Pulumi YAML, now generally available, provides a simple, declarative interface for managing cloud infrastructure, supporting a wide range of programming languages and enabling seamless transitions to more complex coding languages as projects grow. Architecture Templates offer ready-to-use infrastructure setups for common use cases like Kubernetes and serverless applications, facilitating easier deployments across major cloud platforms. The introduction of Project-Level Configuration simplifies the management of complex configurations by allowing predefined settings for different project environments, paving the way for hierarchical configuration and reducing redundancy. These enhancements align with Pulumi's mission to empower cloud engineering teams by integrating software engineering practices into infrastructure management, thereby expanding the accessibility and efficiency of cloud operations.