September 2024 Summaries

The Azure Classic Pulumi Provider's latest major release, version 6.0, incorporates the most recent updates from its upstream Terraform AzureRM provider and introduces new features and improvements from Pulumi. This open-source provider can provision a wide range of Azure resources and is designed to complement the Pulumi Azure Native provider, which offers extensive resource coverage derived directly from official specifications. The release includes enhancements such as an improved diffing strategy in the Pulumi Terraform Bridge, better documentation accuracy through example conversion, and various new modules like Arc Kubernetes, Chaos Studio, and Confidential Ledger, among others. Continuous updates since the last major release have enriched the Pulumi ecosystem, enhancing deployment confidence and documentation quality. Users can access resources such as a Getting Started guide, setup instructions, and how-to guides to facilitate the integration of the Azure Classic provider into their projects, with additional support available through GitHub for those interested in contributing to the provider's development.

Pulumi Docker Containers offer flexibility and convenience for running Pulumi in CI/CD environments without directly installing the CLI, providing several image options including a comprehensive pulumi/pulumi image and slim, language-specific images like pulumi/pulumi-python or pulumi/pulumi-nodejs. These images support versioned language runtimes, allowing users to specify versions for Python, Node.js, and .NET, while Go versions are managed through the Go Toolchains feature. Pulumi Deployments now support setting runtime versions using files like .node-version and .python-version, and the default images will be updated to the latest versions of supported languages by November 2024, ensuring compatibility with new language releases. Additionally, the images come with tools like Poetry and pnpm pre-installed, enhancing the ease of managing dependencies across various environments, whether using Pulumi's CLI, Docker images, GitHub Actions, or Pulumi Deployments.

Serverless Stack (SST) transitioned from AWS's Cloud Development Kit (CDK) to Pulumi to overcome the limitations of being tied to AWS infrastructure and the complexities of using CloudFormation, which hindered debugging and multi-cloud capabilities. Pulumi offers a provider-agnostic infrastructure as code (IaC) solution, allowing SST to support multiple cloud providers and on-premises platforms, thereby enhancing flexibility, visibility, and extensibility for developers. This transition aligns infrastructure code with application development workflows, simplifying the mental model for developers and enabling them to manage infrastructure using the same programming languages as their application code. The move to Pulumi enables SST to improve deployment workflows, integrate seamlessly with application frameworks, and expand provider support, ultimately empowering developers to focus on application development while managing infrastructure more efficiently. SST leverages Pulumi for its deployment engine while maintaining some integrations with Terraform and generates revenue primarily through its managed service, SST Console.

The Pulumi Google Cloud Provider's latest major release, version 8.0, introduces several enhancements, including new features, updated upstream changes, and improved documentation for its open-source platform available on GitHub. Users can now benefit from an enhanced diffing strategy, fixed state upgrades in the Pulumi Terraform Bridge, and improved registry documentation accuracy. Notable new modules include App Hub Application Integration, Blockchain Node Engine, and Managed Kafka, among others. Version 8.0 adds deletion protections for specific resources and introduces a default provisioning label, "goog-pulumi-provisioned," to help users track their Pulumi-provisioned resources in the Google Cloud Platform console. A detailed migration guide is available for users transitioning from version 7 to 8.

At the fourth annual PulumiUP conference, Pulumi unveiled an expanded platform vision that extends beyond infrastructure as code to encompass a comprehensive suite of products designed to automate, secure, and manage cloud operations. The platform now consists of three main products: Pulumi Infrastructure as Code (IaC), which supports coding in any language; Pulumi ESC, a scalable secrets management tool; and Pulumi Insights, which provides visibility and control over infrastructure. These products are supported by core capabilities like Pulumi CrossGuard for policy enforcement, Pulumi Deployments for task orchestration, and Pulumi Copilot, an AI tool for cloud management. Pulumi has grown its customer base to 3,000, including notable companies like Nvidia, BMW, and Unity Games, and continues to improve its offerings with features like enhanced Kubernetes support and integration with Terraform providers. Pulumi Cloud is presented as the most secure and efficient way to adopt these products, with the platform's new capabilities aimed at enabling better collaboration, faster deployment, and more secure, compliant infrastructure management.

Pulumi Environments, Secrets, and Configuration (ESC) is a cutting-edge secrets management and orchestration service designed to address the complexities of managing secrets in a cloud-native environment. Officially launched in October 2023, Pulumi ESC aims to simplify and secure the handling of secrets and configurations by centralizing them across various platforms, such as AWS Secrets Manager and Azure Key Vault, while supporting integration with major tools and cloud providers. Built on an open-source foundation, it offers features like dynamic, short-lived credentials, comprehensive audit logs, and the ability to programmatically access secrets using its SDKs for TypeScript, Python, and Go. Pulumi ESC enhances developer productivity without compromising security, allowing seamless integration into existing workflows and providing a scalable solution for organizations to manage their cloud infrastructure securely. With a flexible, user-friendly pricing model set to begin in 2024, Pulumi ESC is poised to become a vital asset for teams looking to maintain agility while achieving robust security standards.

Pulumi has introduced a new Visual Studio Code (VS Code) Extension aimed at enhancing the productivity of engineers working with Pulumi's Infrastructure as Code (IaC) and Environments, Secrets, and Configuration (ESC) features by integrating these capabilities directly into the IDE. This extension allows users to debug Pulumi applications, manage environments and secrets, and work efficiently with Pulumi YAML by providing functionalities like automatic launch configurations, contextual warnings, and completion lists. The extension supports debugging across multiple programming languages, facilitates editing and managing ESC environments within the IDE, and enhances Pulumi YAML language support with features such as type checking and IntelliSense. Currently in public beta, Pulumi is actively seeking feedback to refine the extension, with plans to introduce further improvements in the coming months to optimize Pulumi development workflows.

Pulumi Insights 2.0 introduces a comprehensive suite of tools designed to enhance the visibility, security, and efficiency of cloud infrastructure, not limited to those managed by Pulumi Infrastructure as Code (IaC). Building upon its predecessor, Insights 2.0 offers new capabilities like Infrastructure Account Scanning, which allows users to scan and manage their entire infrastructure, even those resources not initially under IaC management. The updated platform includes advanced features such as a customizable dashboard, rich visual explorers, structured graphs, and policy violation tracking with auto-remediation options, providing users with detailed insights and control over their infrastructure resources. The integration of Pulumi Copilot and CrossGuard further facilitates intelligent insight discovery and policy compliance, while new UI workflows simplify the process of importing resources into IaC, enabling enterprises to manage their cloud environments more effectively and strategically.

Managing secrets across multi-cloud infrastructures is a challenge that Pulumi seeks to address with its Infrastructure as Code (IaC)-based strategy, which centralizes secret and configuration management within Pulumi's platform. This approach aims to mitigate the issue of "secret sprawl," where sensitive information is dispersed across multiple systems, leading to inconsistencies and security vulnerabilities. By utilizing Pulumi ESC (Environment and Stack Configuration), secrets and configurations are centrally defined and automatically synchronized across various platforms, such as AWS Secrets Manager, Azure Key Vault, and GCP Secrets Manager. Pulumi's method reduces the manual overhead and errors associated with traditional secret management, offering automated updates through scheduled synchronizations or real-time updates via webhooks when secrets are modified. This centralized system enhances security audits and compliance efforts by maintaining a single source of truth for secret management. Pulumi encourages users to explore its documentation and community resources to fully harness the potential of this tool in simplifying and securing secret management processes.

Pulumi has launched its Partner Program to strengthen and formalize relationships with cloud and SaaS companies, aiming to empower partners and accelerate cloud projects for faster time-to-market. This initiative supports nearly 200 existing integrations and focuses on delivering seamless solutions, collaborative go-to-market activities, and tailored support for mutual customers. By leveraging familiar programming languages like Python, TypeScript, .NET, Java, and Go, Pulumi reduces learning curves and enhances productivity, allowing for robust automation and quicker production deployment compared to domain-specific languages. The program includes benefits such as enhanced integration, co-marketing efforts, and access to Pulumi's Infrastructure as Code and Secrets Management capabilities, with partners like Docker, 1Password, and NewRelic joining the initial cohort. Pulumi's strategy is to foster ecosystem growth by connecting partners with new services and technologies, inviting prospective partners to participate through their website.

Pulumi ESC Webhooks is a new feature designed to enhance the management of secrets and configurations by automating workflows and providing real-time notifications, thereby simplifying infrastructure and application updates. It allows development teams to send custom triggers to any URL endpoint and integrates seamlessly with platforms like Slack and Microsoft Teams, enabling real-time notifications and automated actions based on specific events in Pulumi ESC environments. This tool supports hierarchical configurations, ensuring that changes in imported environments trigger the necessary responses, and provides integration options through the Pulumi Cloud Console, Service Provider, and REST API. Pulumi ESC Webhooks aim to improve visibility, streamline workflows, and facilitate automation without the need for custom code, supporting CI/CD pipeline integration and enhancing collaboration among developers, operations teams, and security personnel.

Pulumi has introduced two new features, Projects and Environment Tags, to enhance the management of Pulumi ESC environments. Projects provide a structured way to organize environments into logical groupings, helping large organizations manage environments with distinct names and purposes, thereby reducing conflicts and ensuring changes are made in isolation. Environment Tags allow teams to assign custom tags to environments, streamlining workflows by enabling grouping and searching based on criteria such as teams, project stages, or geographic locations. These enhancements aim to improve the organization, navigation, and collaboration of secrets and configurations within Pulumi ESC, a developer-focused platform for managing secrets and configurations across various cloud environments. The new features are backward compatible, with a transition period to fully adapt to the new system by 2025, and users are encouraged to migrate existing environments to user-created projects to benefit from the new capabilities.

PulumiUP 2024 is set for September 18th and is anticipated to be a key event for cloud professionals, platform engineers, and AI/ML enthusiasts, with over 5,500 participants already registered. This virtual gathering will host a variety of tracks focusing on Cloud and Infrastructure as Code (IaC), Platform Engineering and DevOps, and AI/ML, featuring keynote speeches, panels, and workshops. Attendees will gain insights into the latest innovations and best practices, including discussions on cloud culture, AI governance, and security automation. The event will showcase contributions from industry leaders and organizations like AWS, Google Cloud, Docker, and many others, emphasizing the influence of AI in cloud development and infrastructure. Participants have the opportunity to tailor their experience by selecting sessions that align with their interests and can access live and on-demand content to enhance their understanding of trending topics and Pulumi applications.

Pulumi has introduced a significant update to its ESC Table Editor, now offering full support for provider configurations, thereby enhancing user interaction with application and infrastructure settings. Initially designed to offer a user-friendly interface for managing complex configurations and secrets, the Table Editor allowed CRUD operations on secrets and plaintext values. With the new update, users can now add, edit, and manage provider configurations directly from the table view, streamlining workflows by integrating Pulumi ESC's provider capabilities. Pulumi's provider ecosystem is robust, offering dynamic cloud provider credentials and secure secret retrieval from external managers like AWS Secrets Manager and HashiCorp Vault. This enhances security by eliminating static credentials and enables runtime retrieval of sensitive information to be applied seamlessly to applications. The update simplifies the process by providing a clean, user-friendly form that promotes best practices, reducing the need for external documentation and making the experience more intuitive. Pulumi encourages user feedback to continue refining the Table Editor and other tools based on user needs.

Pulumi's v4.18.0 release enhances the management of Kubernetes resources by improving handling of resource readiness and dependencies, addressing the challenges posed by Kubernetes' eventual consistency model. The update introduces several features, including more informative error reporting for failed resource updates, a "pulumi.com/deletionPropagationPolicy" annotation to optimize resource deletion, and an experimental "pulumi.com/waitFor" annotation for defining custom readiness criteria. These enhancements aim to provide users with greater control and visibility, allowing them to express complex rollout relationships in their preferred programming languages more intuitively. The release also addresses specific issues such as preventing deadlocks in PersistentVolumeClaims and improving readiness handling for Services and DaemonSets. Pulumi encourages feedback from users and offers resources for further exploration through its documentation and community platforms.

Platform engineering is emerging as a critical strategic advantage for cloud-first organizations, enabling faster development cycles and reduced time to market by addressing the complexities of multi-cloud environments. By establishing centralized, consistent, and programmatic processes, platform engineering helps manage deployments, load balancing, and operations efficiently, thereby enhancing productivity, security, and compliance. This approach allows development, DevOps, and infrastructure teams to collaborate seamlessly, resulting in quicker delivery of new features and products. Companies like Aptos Labs, 4IR, and Starburst Data have successfully leveraged platform engineering to significantly reduce their infrastructure deployment times and enhance their competitive edge. As more organizations recognize its transformative potential, platform engineering is becoming a vital component in cloud strategy, offering substantial improvements in deployment and testing times, and leading to faster revenue growth.

Integrating infrastructure data into data warehouses is a complex task for Platform Teams, often hindered by multiple API endpoints, authentication challenges, and inconsistent data formats across cloud providers. The tap-pulumi-cloud connector addresses these issues by automating the process of accessing infrastructure data, utilizing Pulumi Cloud data to avoid custom API integrations and ensuring data consistency. This connector transforms raw data into a structured format ready for analysis, reducing time spent on data wrangling and allowing for quicker insight generation. Built on the Meltano SDK, tap-pulumi-cloud supports various data loaders, enabling seamless data integration into data warehouses like Snowflake, BigQuery, Redshift, or Postgres. It facilitates the creation of analytics such as average resources under management, deployment times, user updates, and can integrate with CI/CD and cost data for comprehensive reporting. The setup process involves installing Meltano, configuring the connector, and running data pipelines, with options for regular orchestration using tools like Airflow or Docker. The system supports upserting data by default, but can be configured to insert new data only, making it adaptable for historical data tracking or current status monitoring.