March 2023 Summaries

Pulumi has introduced a new --teams flag for the pulumi stack init command, allowing developers to assign team permissions directly from the CLI, enhancing the process of managing permissions at scale. This feature, available with Pulumi v3.59.0, complements existing methods like the Pulumi Service REST API and Pulumi Service Provider by enabling developers to initialize stacks with the appropriate permissions without leaving the CLI. This update addresses feedback from users who needed more efficient ways to handle permissions for multiple stacks, particularly in organizations using teams. The --teams flag, which can be used multiple times to assign access to different teams, simplifies the process by eliminating the need to use the Pulumi Service console for assigning team access, thus saving time and improving productivity. Currently, the feature grants read and write access, which is the most commonly needed level of permission. Pulumi aims to continuously enhance cloud productivity by incorporating user feedback and improving the ergonomics of their tools.

In the fourth installment of a blog series on Zephyr Archaeotech Emporium's use of Pulumi, the focus is on employing Stack References to link different projects and enhance infrastructure management. Zephyr, a fictional company, has structured its Pulumi projects into three distinct parts—zephyr-infra for base infrastructure, zephyr-k8s for the Kubernetes platform, and zephyr-app for deploying online store services. This structure supports multiple teams and allows the application to evolve separately from its supporting infrastructure. The blog elaborates on Stack References, a method allowing different stacks to share information, addressing cross-project dependencies without hard-coding values. Best practices include exporting only necessary stack outputs, minimizing the use of stack references to reduce project coupling, and parameterizing stack references to avoid hardcoding. The blog also highlights the importance of planning for cascading stack updates and keeping security considerations in mind. The series aims to gradually expose best practices in infrastructure as code (IaC) management, and the next post will address security and role-based access control (RBAC).

Pulumi Deployments introduces a modern approach to managing and deploying cloud infrastructure at scale, addressing limitations of traditional CI/CD workflows that are not optimized for infrastructure as a product. This fully managed platform provides a robust set of tools, including secure and elastic compute, centralized configuration, and REST API-driven deployment triggers, enabling users to automate infrastructure updates dynamically and efficiently. With a focus on programmability, Pulumi Deployments allows for the creation of custom infrastructure APIs, leveraging RESTful interfaces to facilitate on-demand infrastructure creation, updates, and management. The platform's features, such as the Pulumi Service UI and Remote Automation API, offer flexibility in configuring deployment settings and triggers, empowering enterprises to build scalable, secure, and customizable cloud solutions without the constraints of static YAML configurations. By combining Deployment Settings and Triggers, users can craft tailored deployment pathways that align with their organizational requirements, paving the way for innovative infrastructure delivery and management.

Pulumi has introduced a new method called StackReference.getOutputDetails, which allows users to directly access output values from other stacks without needing to call Output.apply repeatedly. Stack references are a feature in Pulumi that enable one stack to access output values from another, such as an application stack retrieving a VPC ID from a network stack. Previously, using the getOutput method returned an Output value that required the use of apply for transformations, but the new getOutputDetails method returns a plain value, simplifying the process. This method also extends to secret outputs, allowing access to sensitive information like database passwords. However, it is limited to stack references where the referenced stack has already been deployed, and cannot be used for all Output values, which only become available during the pulumi update process. This feature is available in all stable Pulumi SDKs starting from version 3.55.0.

Pulumi has introduced various improvements across its engineering areas in January and February, focusing on enhancing user experience and functionality. Key updates include colorized stack traces for Node.js errors in the CLI, the introduction of a --shell option for stack outputs, and support for plugins hosted on GitLab. The Language SDKs saw enhancements such as helper functions for JSON serialization and deserialization, and the Python SDK now supports native arm64 binaries. In the Automation API, users can now programmatically manage stack tags and clone from Azure DevOps. Pulumi has also strengthened its cloud provider offerings with new resources for Azure and AWS, and added community packages to its Registry. Pulumi Deployments now integrate with OpenID Connect (OIDC) to provide temporary credentials and granular access control, improving security and compliance. These updates aim to streamline workflows, improve performance, and enhance security for Pulumi users.

Pulumi's approach to reducing error rates involves a meticulous process of reading every error message generated by their API, which has led to a 17-fold year-over-year reduction in errors. This strategy challenges the conventional reliance on aggregate error views and sophisticated observability tools, advocating instead for direct engagement with error messages to enhance system reliability. By prioritizing the review of 5XX errors within a Slack channel, Pulumi ensures that each error is promptly addressed by the on-call engineer, promoting a culture of accountability and continuous improvement. The underlying principle is that as API traffic increases, the error rate must decrease to maintain operational efficiency within the fixed capacity of on-call resources. This process not only improves system reliability but also fosters a deeper understanding of user experience and product development needs, as the team becomes more attuned to customer requirements and potential scaling challenges. Despite its limitations at massive scales, such as Google's, Pulumi's strategy underscores the importance of tailoring processes to the current scale of operations, allowing for a high-performance culture focused on customer satisfaction and proactive problem-solving.

Zephyr Archaeotech Emporium, a fictional company, has been using Pulumi to manage its online retail store, and as the company has grown, its approach to Pulumi has evolved to accommodate its expanding infrastructure and team structure. Initially, Zephyr used a single Pulumi project, but organizational growth prompted a switch to multiple projects to reflect their needs better and allow for more effective management of resources. Several factors influenced this decision, including team and department structure, security, resource relationships, lifecycle, change rate, and the desire to minimize the "blast radius" of potential errors. Ultimately, Zephyr restructured its Pulumi usage into three separate projects: infrastructure, platform, and application, each housed in its own GitHub repository, thereby aligning with best practices that offer flexibility and accommodate the evolving landscape of their operations. This restructuring allows different teams to manage different infrastructure aspects, providing flexibility at the cost of increased complexity, similar to the trade-offs between monolithic and microservice architectures.

Point-and-click methods for managing cloud resources can become cumbersome as organizations grow, necessitating a transition to Infrastructure as Code (IaC) solutions like Pulumi, which offers a robust import process for existing resources. This article outlines a method for automating the bulk importation of Google Cloud resources into Pulumi using Google's Config Connector to query resources and output them as YAML files, which are then transformed into JSON suitable for Pulumi import operations via a Python script. The process involves mapping Kubernetes manifests to Pulumi resource types and ensuring resource names are unique to avoid import errors. The article also highlights the importance of managing cloud resources with code-centric approaches and provides guidance on setting up CI/CD pipelines and implementing policy as code for compliance and security. Pulumi's import feature facilitates the transition to IaC by enabling organizations to manage their Google Cloud resources more efficiently, even when resources were initially created through manual methods.

Zephyr, a fictional company managing its online retail store with Pulumi, explores best practices for enabling multiple developers to collaborate on projects and efficiently use Git branching. Initially managing their infrastructure with a two-stack approach (development and production), the team faces challenges as more developers work concurrently, leading to potential deployment conflicts. To address this, they shift to allowing each developer to create personal Pulumi stacks, enabling them to iterate and deploy independently in a shared cloud environment without disrupting others. This strategy also includes maintaining a feature-branch workflow with frequent merges to a main branch, ensuring a consistent integration environment. As a result, the team can test changes in a near-production setup before full deployment, reducing risks and improving development velocity. This approach highlights the importance of agile practices in managing complex, containerized applications as the Zephyr team continues to refine their use of Pulumi.

International Women's Day highlights the theme of #EmbraceEquity, emphasizing the creation of equitable environments in the workplace, where opportunities are tailored to individual needs irrespective of gender, ethnicity, or background. Pulumi exemplifies this by fostering an inclusive space where diverse experiences, including those of women, men, and nonbinary individuals, contribute to technological innovation. Despite women's significant contributions, challenges persist, such as the gender pay gap, which remains between 13% and 17%, and the disproportionate assignment of administrative tasks to women. Encouragement from role models and mentors is vital for women in tech, as shown by Logitech and Girls Who Code's findings. Pulumi showcases the experiences of its female tech professionals, highlighting their journeys, motivations, and advice for thriving in the industry, such as embracing change, setting personal boundaries, and relying on a supportive female network. These narratives underscore the importance of diverse experiences in fostering innovation and the ongoing need for equitable opportunities in the tech sector.

The Pulumi Docker Provider, a leading tool since 2018, allows users to manage Docker resources such as containers and images through Pulumi programs, with its popular docker.Image resource being enhanced in version 4 to improve performance and user experience. The latest updates, driven by community feedback, include significantly faster build times via reduced rebuilds and enhanced BuildKit support, facilitating cross-platform builds, richer Docker build logs in Pulumi output, and expanded support for Pulumi YAML and Java. The version 4 release, inspired by contributions from the Materialize team, showcases these features through a demo involving a MERN stack application, highlighting ease of use and deployment speed. The updates also include a streamlined migration process despite some breaking changes, ensuring a smooth transition to the new version while maintaining support for all Pulumi languages, enabling CRUD lifecycle participation and enhanced caching options for better performance.

Speeding up Docker image builds with Pulumi involves leveraging Docker's BuildKit, using registry or layer caches to reuse work, and writing multi-stage Dockerfiles to minimize production image sizes by skipping build-time dependencies. Pulumi offers two providers for Docker image builds: the original Docker provider and the newer Docker Build provider. The Docker Build provider supports advanced build features like multi-platform images, build secrets, and Docker Build Cloud, making it suitable for new projects. The key to optimizing build times is to ensure persistent caching through registry configurations and to organize Dockerfile layers effectively, placing less volatile layers first to avoid unnecessary rebuilds. Multi-stage builds allow for separating development tools from runtime environments, thereby reducing image sizes and ensuring fast builds, especially in CI environments. Pulumi's declarative approach helps maintain these optimizations across multiple builds, resulting in significant time savings.