December 2024 Summaries

Pulumi's updates in the latter half of 2024 introduced significant enhancements across its cloud engineering platform, focusing on infrastructure as code (IaC) and integrations with major cloud providers. Noteworthy developments include the Pulumi Visual Studio Code Extension for improved IaC debugging, Pulumi EKS 3.0 with enhanced security and support for Amazon Linux 2023, and the Pulumi Kubernetes Operator 2.0 offering a scalable architecture and improved customization. Pulumi Insights 2.0 introduced a revamped Resource Explorer, while the platform expanded its ecosystem with compatibility for AWS CDK, Azure v6, and Google Cloud Provider v8.0.0. Pulumi Copilot saw enhancements like REST API integration and organization-level customization, while Pulumi ESC reached general availability with advanced secrets management and real-time notifications. Deployment enhancements included dependency caching and a Kubernetes-native agent, aiming to streamline and accelerate cloud resource management. Pulumi Cloud Console improvements focused on compliance visibility and performance, further solidifying Pulumi’s role in cloud engineering innovation.

Pulumi had a transformative year in 2024, unveiling a new vision with a comprehensive platform that includes Pulumi IaC, Pulumi ESC, and Pulumi Insights, expanding beyond infrastructure as code to offer centralized secrets management and enhanced visibility over infrastructure resources. Over 3,000 customers, including notable companies like Nvidia and BMW, have adopted Pulumi's solutions for improved cloud automation, security, and management, with significant updates such as Pulumi Insights 2.0 and the introduction of Pulumi Copilot leveraging AI for infrastructure management. Pulumi's community also saw substantial growth, with contributions soaring and user groups expanding globally, while the company's innovations led to notable achievements like 71% cloud cost savings for some customers. As Pulumi looks ahead to 2025, it plans to continue its growth trajectory and further innovate in cloud infrastructure and security.

The blog post explores the use of the Secrets Store CSI Driver in Kubernetes for managing secrets securely without storing them as native Kubernetes secrets. It highlights the limitations of using the External Secrets Operator, which still involves creating Kubernetes secrets, and introduces the Secrets Store CSI Driver as a solution for mounting secrets directly into pods from external secret stores. The post explains the architecture and deployment process of the CSI Driver using Pulumi, emphasizing its benefits in managing secret lifecycles outside of Kubernetes while maintaining a native experience. It also compares the External Secrets Operator and the Secrets Store CSI Driver, noting their primary use cases, integration capabilities, mechanisms for secret delivery, and security considerations. The CSI Driver is recommended for scenarios requiring direct access to secrets without intermediate storage in Kubernetes secrets, while the External Secrets Operator is suited for accessing secrets as Kubernetes native secrets.

Pulumi has announced the integration of AWS Systems Manager - Parameter Store within its Environments, Secrets, and Configuration (ESC), allowing development teams to securely and conveniently import AWS Parameter Store parameters as environment variables or Pulumi Config in their workflows. This integration, which has been highly requested by the community, simplifies the management of secrets and configurations by consolidating them into a single source. The process involves utilizing AWS login and defining path names to retrieve parameters, which are then used in Pulumi ESC environments to run commands without storing credentials locally. The integration expands Pulumi's roster of supported services, offering developers greater flexibility and streamlining their workflows, while also inviting feedback to improve future solutions.

Amazon EKS Auto Mode is a new feature from AWS that automates infrastructure management for Kubernetes clusters, simplifying tasks like compute, storage, and networking by leveraging AWS's operational expertise. This enhancement allows for automatic provisioning and scaling of EC2 instances, streamlines block storage setup, and manages networking components like load balancers, thereby reducing the operational overhead for teams. Pulumi supports EKS Auto Mode, enabling users to manage these processes through infrastructure as code across various AWS providers. This allows teams to focus more on application development rather than infrastructure maintenance. By enabling EKS Auto Mode, users can deploy applications more efficiently, as AWS handles complex tasks such as cluster autoscaling, storage management, and network configuration. Pulumi's integration further allows developers to use their preferred programming languages to define and manage these automated clusters, providing a streamlined development and deployment process for production-grade Kubernetes workloads.

Simen A. W. Olsen shares insights on the pitfalls of designing overly complex, future-proof system architectures, emphasizing the importance of adaptability and simplicity in system design. Drawing on personal experiences, Olsen reflects on a past project where an initially celebrated architectural plan quickly became a hindrance due to its excessive focus on anticipating every future need. He highlights the common mistake of over-planning, even in agile environments, and underscores the success of companies like Netflix, which opted for simpler, adaptable solutions over perfect ones. Olsen identifies three core principles for creating change-ready architectures: embracing simplicity, making change inexpensive through tools like automated testing and continuous deployment, and learning through real-world feedback rather than predictions. He argues that the cost of changing a simple system is typically lower than maintaining an over-engineered one, advocating for systems that are easy to modify rather than those with supposed future-proof features.

Pulumi ESC enhances the management of secrets and configuration across multiple environments by introducing composable environments that can import other environments, thus reducing duplication and complexity. This system centralizes the management of secrets and configurations, allowing them to be reused across Pulumi Infrastructure as Code (IaC) Stacks, Pulumi Insights Accounts, and various platforms through integrations. The recent updates have introduced discoverability features, including the "Imported By" tab, which enables users to understand downstream dependencies by showing all environments and stacks that import a specific environment. These features allow users to filter and sort by imported versions to assess the impact of changes and provide warnings for potential disruptions when deleting environments or modifying version tags. Additionally, the enhanced environment page and stack visibility offer valuable context about relationships and dependencies, making it easier to manage complex infrastructures with confidence.

Pulumi Copilot, initially designed to automate cloud management using large language models, has been widely adopted by customers for enhanced cloud infrastructure management. Recently, Pulumi announced the Pulumi Copilot REST API, allowing users to integrate AI-driven infrastructure management into their own platforms and applications. This API, while still in preview, offers capabilities such as embedding Copilot functions into various tools, supporting multi-user interactions in collaboration platforms, automating queries based on events, and enabling mobile client access. It requires specific configuration steps, including setting environment variables and understanding the necessary cloud context parameters for effective API interactions. The API also supports multi-turn conversations for maintaining context across queries. Pulumi is actively seeking user feedback to refine its offerings as they progress in their AI initiatives.

Pulumi's development of the AI assistant, Pulumi Copilot, for cloud infrastructure highlights the challenges and insights gained from integrating large language models (LLMs) into software engineering. A significant challenge arose when Copilot suggested a non-existent command, revealing the need to balance prompt engineering with traditional coding. This experience underscored the importance of using LLMs for natural language tasks while relying on traditional code for structured data transformations to reduce costs and improve efficiency. Pulumi refined Copilot by breaking it into modular "skills" that handle specific tasks, enhancing the AI's ability to interact with users and manage infrastructure queries. Despite the potential for AI-generated outputs to appear accurate while containing errors, Pulumi developed rigorous testing methods, including using LLMs as evaluators, to ensure response accuracy. Interestingly, AI hallucinations sometimes revealed user expectations, offering unexpected insights into product development. These lessons led to the launch of the Pulumi Copilot REST API, allowing users to integrate Copilot's capabilities into their own tools, with a focus on continually learning from user interactions to refine AI systems.

Platform engineering is becoming a critical function within mid-to-large organizations, moving beyond the DevOps trend by focusing on creating self-service tools for developers while ensuring security, compliance, and reliability through DevSecOps practices. Experts at PulumiUP Europe 2024 highlighted core pillars of successful platform engineering which include developer experience, reliability, scalability, security, compliance, automation, and observability. They emphasized the importance of aligning DevOps and security, promoting a culture of collaboration, and addressing challenges such as balancing autonomy and control, driving adoption, and adapting to evolving needs. The panel stressed the need for integrating security early in the development process, automating security measures, and using observability tools to enhance platform health and performance. As platform engineering continues to evolve, its role in supporting scalable and secure development environments will become even more crucial, with success measured through a combination of technical, operational, and business metrics.

Kubernetes Operators serve as a key component in automating complex operational tasks and managing the lifecycle of applications on Kubernetes-powered platforms. These operators extend Kubernetes' capabilities by utilizing Custom Resource Definitions (CRDs) to enable advanced automation, focusing on specific domains such as databases, storage, and CI/CD pipelines. Unlike general-purpose controllers that manage cluster resources, operators provide tailored solutions for application-specific operations like backups, scaling, and error handling. They offer a higher level of automation and modularity, allowing platform engineering teams to encode operational knowledge into automated processes, thereby enhancing productivity and ensuring consistent deployments across environments. Advanced operators, like the CloudNativePG for PostgreSQL, Flux for GitOps deployment, and Strimzi for Apache Kafka, exemplify the benefits of using operators to simplify the deployment and management of complex services, ultimately leading to more reliable, efficient, and observable platforms.

Pulumi ESC has introduced native support for secrets rotation, simplifying secrets lifecycle management, particularly for static secrets like database passwords and API keys, which require regular rotation to maintain security. The system employs a custom declarative configuration to automate secret rotation schedules through Pulumi ESC and IaC integration, allowing for seamless transitions with zero downtime by maintaining two versions—current and previous—of each secret. This solution includes a generic Rotator component that manages credential lifecycles and a Scheduler component that orchestrates rotations based on changes in the ESC environment configuration. The approach enables dynamic retrieval of updated credentials, ensuring applications automatically receive the latest credentials without additional configuration, thus offering a scalable and auditable solution that can be easily applied across multiple environments.

Infrastructure as Code (IaC) has transformed cloud resource management, offering scalability and efficiency, and Pulumi stands out by enabling users to program cloud infrastructure using popular programming languages. Pulumi Cloud, available as a SaaS or self-hosted solution, provides a comprehensive management platform that automates deployments, centralizes secrets management, and utilizes AI for infrastructure oversight, contrasting with the do-it-yourself (DIY) approach that demands significant engineering and operational resources. Pulumi Cloud ensures robust security and compliance, integrating seamlessly with identity providers for role-based access control, and offers features like Pulumi Insights for infrastructure analytics and AI-driven insights. It supports organizational scaling by simplifying onboarding with standardized processes and built-in role-based access control, freeing up valuable engineering resources. As a fully managed service, Pulumi Cloud minimizes operational burdens, reduces the risk of outages, enhances security, and ultimately leads to cost savings and increased productivity, allowing organizations to focus on innovation rather than backend management intricacies.

Containers have become ubiquitous, running on everything from quantum computers to smart toasters, with a wide array of deployment options ranging from mainstream cloud providers to experimental setups. This exploration reveals the vast and unexpected landscape of container orchestration, highlighting common platforms such as Amazon EKS and Google Kubernetes Engine, as well as more niche and regional options like those offered by Huawei and Tencent. It delves into specialized solutions for edge computing and serverless functions, and even the most unconventional methods for deploying containers, emphasizing the adaptability and complexity of modern container ecosystems. Pulumi is suggested as a tool to manage this diversity, providing a unified approach to deploying and orchestrating containers across various environments.

Pulumi Insights 2.0 introduces Insights Account Discovery, a tool designed to provide comprehensive visibility and management of complex enterprise cloud infrastructures. It allows users to set up and scan all infrastructure accounts through a simple UI, creating child accounts for different groups and regions, which improves control over discovery behavior. The product integrates Pulumi ESC for secure credential management and supports Infrastructure as Code resources, enabling logical grouping and resource exploration through the Insights Resources Explorer and AI-powered Resource Search. This functionality facilitates easier access to critical infrastructure information, allowing users to ask specific questions and efficiently manage their resources. While currently free during its public preview phase, Pulumi Insights plans to introduce a tiered pricing model in early 2025, and additional features such as Pulumi Crossguard Policy support are anticipated.

Pulumi has announced the 1.0 release of its pulumi-cdk library for TypeScript, which enhances compatibility with AWS Cloud Development Kit (CDK) constructs, allowing for the deployment of almost any CDK construct with Pulumi. This release addresses limitations present in the preview version and expands the interoperability between Pulumi and the AWS CDK ecosystem. Now, Pulumi users can confidently use over 1600 constructs from AWS's construct hub, while AWS CDK users can transition from CloudFormation to Pulumi more easily, benefiting from Pulumi's features like Policy as Code, Audit Logs, and the ability to manage resources across various cloud service providers. The pulumi-cdk 1.0 version supports key CDK features such as assets, custom resources, nested stacks, and aspects, eliminating the need for a cdk bootstrap before running a pulumi up. The project aims to expand support beyond TypeScript and explore options for seamless migration of existing CDK applications deployed with CloudFormation. Pulumi and AWS CDK's integration offers a robust platform for cloud engineering, fostering a collaborative ecosystem for developers to build and manage cloud infrastructure efficiently.

Pulumi offers a suite of tools designed to streamline the management of AWS cloud infrastructure by leveraging infrastructure as code (IaC) and other advanced capabilities. Initially founded in 2017 to address the growing complexities of cloud applications, Pulumi allows teams to automate infrastructure, manage secrets, and ensure compliance through AI-assisted insights. The company has developed various innovations, such as Pulumi Crosswalk for AWS, which simplifies infrastructure tasks using well-architected best practices, and Pulumi CrossGuard, which enforces cloud governance. Pulumi also supports AWS Cloud Development Kit (CDK) constructs, allowing integration with Pulumi's resources. Additionally, Pulumi Insights provides visibility and optimization for cloud infrastructure, while Pulumi ESC offers centralized secrets management, addressing security challenges. These offerings collectively enhance productivity, collaboration, and security for organizations managing modern cloud environments, with Pulumi continuously expanding its capabilities to meet evolving demands.

Amazon Web Services (AWS) has launched Amazon EKS Hybrid Nodes, a new feature in its Amazon Elastic Kubernetes Service (EKS) portfolio, which allows organizations to maintain their Kubernetes control plane on AWS while running workloads on-premises or at the edge. This hybrid approach provides the benefits of AWS's scalable control plane management combined with the flexibility to run certain workloads locally, addressing needs related to data sovereignty, latency, and existing infrastructure investments. Pulumi, an EKS Service Ready Partner, supports this feature by enabling infrastructure management through its infrastructure as code platform, which allows teams to manage EKS clusters and Kubernetes resources using familiar programming languages. Pulumi's platform also provides secure management of credentials and secrets across hybrid environments and supports automation and operations through its CI/CD capabilities. This integration helps organizations effectively manage hybrid Kubernetes deployments across various environments, including cloud and on-premises setups, while maintaining security and infrastructure automation.