October 2023 Summaries

Infrastructure as Code (IaC) is a transformative approach that allows engineers to automate cloud infrastructure management by writing code rather than performing manual tasks, thereby reducing error and increasing scalability. This method is increasingly essential as modern applications, whether simple or complex, rely on diverse cloud infrastructures, necessitating consistent management and adaptation to evolving needs. The text introduces the concept and benefits of IaC, emphasizing its capacity to encode infrastructure configurations and changes in code, which can be reviewed, versioned, and replicated across different environments. It highlights the role of declarative code in IaC, which allows for the expression of desired infrastructure states and the automatic orchestration of necessary changes by IaC tools. Pulumi is presented as a noteworthy tool in this space, supporting various programming languages and cloud services, and offering a sophisticated "Automation API" for integrating IaC workflows into larger software projects. The series promises to explore the practical setup of IaC using Pulumi in its subsequent parts.

Pulumi ESC (Environments, Secrets, and Configuration) is a newly released service designed to streamline the management of cloud configurations and secrets, offering a centralized approach to handle these critical aspects of cloud development. This service, akin to a Swiss Army knife for cloud infrastructure, aims to simplify credentials management, particularly with its esc run functionality, which allows users to execute cloud commands without manually configuring credentials locally. By dynamically generating short-term, scoped credentials via OpenID Connect (OIDC), Pulumi ESC enhances security by eliminating the local storage of credentials, thus minimizing exposure risks. The service also facilitates streamlined collaboration by providing a consistent environment for team members, making it easier to manage multiple development, staging, and production environments. Pulumi ESC can be integrated into various real-world scenarios, including automated data backup, disaster recovery, environment management, and CI/CD pipelines, ensuring efficient, secure, and error-free cloud operations.

LangChain, an open-source Python framework, is revolutionizing DevOps by facilitating the integration of advanced AI models like OpenAI’s GPT, Google’s Gemini, and Meta’s LLaMA into software development. Highlighted at the "Generative AI, Apps, and DevOps | AI/ML Talks" event in Seattle, this tool is gaining popularity for enabling developers to build sophisticated generative AI applications, such as retrieval-augmented chatbots, with ease. Patrick Debois, known for coining "DevOps," is leveraging LangChain to enhance DevOps through AI-powered solutions, enabling automation, resource optimization, and seamless integration of machine learning models. LangChain's capabilities extend to real-world applications, including monitoring, anomaly detection, and improving developer productivity by providing actionable insights and enhancing security practices. As AI continues to shape the future of software engineering, LangChain positions itself as a key player in accelerating the development of AI-powered DevOps solutions.

Pulumi has enhanced its capabilities to support importing resources directly from Terraform state files, enabling a smoother transition for users with existing infrastructure. Previously, Pulumi provided AWS-specific code for importing from .tfstate files, but this was limited to Go or TypeScript applications. With the release of Pulumi CLI v3.87.0, the converter system now includes state alongside code, allowing users to import resources from Terraform by using commands to set up a Pulumi stack and import from a terraform.tfstate file. The tool attempts to convert the resources into corresponding Pulumi types and import strings, though some resources may require manual adjustments if the automated process encounters issues. Users are encouraged to report any errors to help improve the system, and the new functionality can be accessed by downloading the latest version of the Pulumi CLI.

Pulumi Deployments, now generally available, is a fully managed platform designed to simplify and accelerate cloud infrastructure management for organizations at any stage of their cloud journey. It offers a robust set of features including scalability, automation, and integration capabilities, allowing users to execute various infrastructure as code actions directly within Pulumi’s managed service. With enhancements such as OpenID Connect configuration, scalability improvements, review stacks, and custom deployment webhook events, Pulumi Deployments has seen considerable adoption, executing hundreds of thousands of deployments. New features include a pricing model effective November 1st, customer-managed deployment agents for on-premises needs, and GitHub Enterprise support. The platform also introduces an organization-level deployments page for streamlined management across multiple teams. Pulumi Deployments aims to provide a comprehensive solution for cloud engineers and platform teams to efficiently transition from code to cloud, offering both ease of setup and integration with existing version control systems.

Pulumi's CrossGuard policy engine offers a flexible approach to enforcing infrastructure compliance through policies that can issue warnings or errors when violations occur. Recently, Pulumi introduced remediation policies, which not only detect policy violations but also automatically correct them, ensuring that deployments adhere to specified guidelines without user intervention. This feature supports a range of use cases, from tagging resources to enforcing security measures, and can be configured to apply organization-wide or to specific stacks. Remediation policies leverage Pulumi's policy-as-code capabilities, allowing developers to use languages like JavaScript, Python, and TypeScript to write custom policies. These policies can be run locally or published to Pulumi Cloud for broader enforcement. The remediation feature is inspired by static analysis and code-fixing techniques, enhancing compliance and productivity by automating corrections while maintaining transparency about changes. This new capability is available in Pulumi's open-source SDK, making it accessible for users to implement and experiment with policy packs that automatically remediate infrastructure issues.

Pulumi Cloud experienced a 24-minute outage on October 6, 2023, due to a database migration error that resulted in a significant portion of requests not being processed. The migration involved modifying foreign keys on a table, but the pre-production testing did not accurately simulate the production environment's traffic load, leading to a table lock that starved the database of connections. This incident highlighted the importance of accounting for even low-traffic tables in scaling considerations, as the block on a less busy table still caused significant disruption. In response, Pulumi has conducted a postmortem and is implementing improvements in its Software Development Lifecycle, including enforcing safe SQL constraints, enhancing pre-production testing fidelity, and creating new tools for testing database performance. Additionally, Pulumi plans to split its service into separate failure domains to preserve core workload functionality during similar incidents. Pulumi emphasizes its commitment to transparency and operational excellence, apologizing to users for the disruption and pledging to learn and improve from this experience.

Pulumi has announced significant developments, including the launch of two new products, Pulumi ESC and Pulumi for Platform Teams, alongside securing Series C funding. Pulumi ESC offers a novel approach to managing cloud environments, secrets, and configurations, enhancing security and scalability across various applications and infrastructure. Pulumi for Platform Teams leverages insights from over 2,000 customers to provide streamlined patterns and practices, such as a developer portal and compliance policy management, aimed at improving internal developer platforms. These initiatives are part of Pulumi's broader strategy to innovate in infrastructure as code, following the launch of Pulumi Insights, which utilizes generative AI to enhance cloud management. The Series C funding will support the expansion of Pulumi's team and the continuous reimagining of cloud programming solutions. Pulumi invites users to engage with their new offerings, provide feedback, and explore career opportunities within the company.

Pulumi has announced a new suite of tools to facilitate the creation of internal developer portals, aiming to streamline infrastructure management for organizations of varying sizes. These tools, which include the New Project Wizard, Organization Templates, and the Pulumi Backstage Plugin, enhance Pulumi's offerings by allowing developers to deploy infrastructure via point-and-click interfaces, standardize infrastructure creation, and integrate with popular developer platforms like Backstage. Customers can now use private templates and leverage Pulumi's Automation API and Deployments to customize their developer portals, providing a unified experience that reduces complexity and increases productivity. This development is aimed at both simplifying the self-service provisioning of infrastructure and enhancing collaboration and consistency across organizations, allowing platform teams to share templates and configurations easily. Pulumi's new features are designed to integrate seamlessly with existing workflows, offering extensibility and adaptability for diverse development environments.

Pulumi has witnessed increased adoption by platform teams over the past two years, leading to the introduction of new features in Pulumi Cloud to enhance the entire lifecycle of cloud delivery. These include the launch of developer portals to expose reusable cloud infrastructure components, integration with Backstage via a new Pulumi plugin, and the introduction of organization templates and a new project wizard for easier infrastructure deployment. Pulumi has also expanded its CrossGuard policy-as-code framework with compliance-ready and remediation policies to enforce and automatically rectify compliance standards across various cloud providers. Additionally, Pulumi Deployments has moved to general availability, offering a unified view of deployments, integration with GitHub Enterprise, and plans for self-hosted deployment agents to cater to unique compliance or security needs. These enhancements aim to empower platform teams by providing robust infrastructure management, policy enforcement, and seamless deployment capabilities within Pulumi Cloud.

Pulumi ESC, a new product from Pulumi, is designed to manage and streamline secrets and configuration complexities across cloud infrastructure and application environments. It introduces a novel configuration-as-code approach, allowing teams to aggregate and manage secrets and configurations from various sources, and consume them across different services. Pulumi ESC integrates seamlessly with Pulumi Infrastructure as Code (IaC) but can also function independently, offering hierarchical and composable environment management. This tool supports dynamic configuration providers like AWS Secrets Manager, Azure KeyVault, and Google Cloud OIDC, among others, providing a unified interface for configuration and secrets management. Its CLI, REST API, and integration with Pulumi IaC enable environments to be accessed from any application or infrastructure provider, enhancing configuration management with audit logs and role-based access control. The open-source project supports flexible, code-based configuration management, addressing challenges such as secrets sprawl, duplication, and long-lived static secrets, and is available as a fully managed service in Pulumi Cloud. Pulumi ESC aims to simplify secrets and configuration management, offering dynamic and secure solutions across different cloud providers, with plans for further enhancements and integrations.

Backstage is an open-source framework developed by Spotify and contributed to the CNCF, designed to create developer portals that unify infrastructure tools, templates, services, and documentation. The introduction of the Pulumi Plugin for Backstage allows developers to integrate infrastructure management seamlessly, providing functionalities such as creating new Pulumi projects from organization templates, running previews and updates, and viewing deployments. The Pulumi Backstage integration comprises two main components: the Pulumi Scaffolder Backend Module, which extends Backstage's scaffolding capabilities with actions like pulumi:new and pulumi:up, and the Pulumi Plugin, which displays critical details and aggregates information from the Pulumi Cloud. This integration streamlines infrastructure management, enhances the developer experience, and fosters a DevOps culture by offering benefits like a consolidated view of infrastructure, simplified navigation, transparent communication, and knowledge sharing. It empowers developers with self-service operations, accelerates development, and promotes standardized workflows while adopting best practices in infrastructure management and application deployment. The Pulumi Backstage Plugin is available through a GitHub repository for self-hosted Backstage or through the Roadie managed service.

Pulumi has announced a $41 million Series C funding round from Madrona Ventures, NEA, Tola Capital, and Strike Capital to fuel further innovation in infrastructure as code technology and expand into new product areas. The company has gained significant traction, with over 2,000 customers, including more than half of the Fortune 50, and a community of over 150,000 end users. Pulumi's unique approach allows developers to use any programming language for infrastructure as code, providing a more flexible and integrated experience that extends beyond traditional infrastructure management. This approach has led to the development of additional tools such as Policy as Code, Testing and CI/CD Automation, and Pulumi Deployments, which offer features like server-side orchestration and AI-driven insights. As Pulumi evolves into a comprehensive enterprise platform, it remains committed to open source principles and aims to democratize cloud access while focusing on security, developer experience, and AI integration. The new funding will support ongoing efforts to enhance the product and expand the community, continuing Pulumi's mission to transform cloud infrastructure management.