February 2021 Summaries

Andy Davies, a Senior Software Developer at Reaktor, discusses using the Pulumi Automation API to enhance observability in infrastructure provisioning, particularly for applications that create virtual private clouds (VPCs) and store credentials in a Vault instance. By integrating Honeycomb's Go Beeline, Davies captures valuable data such as durations and errors, aiding in the identification of performance bottlenecks. This approach involves modifying the progress streams to track resource creation in real-time, offering insights into the highly parallelized nature of Pulumi processes and identifying time-consuming tasks like NAT Gateway creation. The implementation is described as relatively straightforward and a significant productivity booster compared to traditional methods, with future plans to expand the observability framework to include more detailed information about resource operations and failures.

Pulumi, in conjunction with Aiven's services, provides a framework for creating cloud-agnostic infrastructure that can be migrated seamlessly across different cloud environments without downtime. This tutorial demonstrates using Python, Pulumi, Grafana, and AWS Lambda to simulate recording temperature data in an InfluxDB database, showcasing Pulumi's ability to manage infrastructure with a single command. Aiven facilitates the deployment of fully-managed open-source cloud databases like InfluxDB, enabling accelerated application development across AWS, GCP, and Azure. The process involves setting up required accounts and configurations, deploying services in Aiven, and migrating them across clouds while maintaining data flow, which is visualized in Grafana. This approach is applicable to various Aiven services, offering flexibility and security in cloud migrations. The tutorial underscores the use of infrastructure as code to enable zero-downtime service migrations, with the potential for broader application in managing cloud-based data infrastructure.

The Pulumi Service is a platform that aids teams in delivering and managing cloud applications and infrastructure by providing a comprehensive view of the infrastructure through a unified console. It integrates with major Identity Providers (IdPs) such as Okta, Azure Active Directory, and Google Cloud Identity & Access Management, enabling Single Sign-On (SSO) with SAML and user and group synchronization via SCIM 2.0, which streamlines access management and enhances security. By leveraging these integrations, organizations can efficiently manage user access and permissions from a central point, simplifying identity and access management workflows. The Pulumi Service supports SAML 2.0 for SSO, allowing IT departments to manage employee access efficiently, and SCIM for synchronizing users and groups, ensuring that team members have appropriate permissions for managing infrastructure. Existing Pulumi customers can configure these features through their organization settings, while new users are encouraged to start a trial or request a demo to explore the platform's capabilities.

Unauthorized access to infrastructure can lead to severe consequences, such as data breaches and exploitation of security vulnerabilities, highlighting the importance of protecting infrastructure secrets like passwords and access tokens. Pulumi addresses these risks by providing a secure-by-default platform with built-in secret storage that safeguards all secret data using encryption, whether users opt for the hosted Pulumi Service, the Self-Hosted Pulumi Service, or a self-managed backend. This platform automatically encrypts state metadata, including both non-secret and secret information, ensuring that secret values are never exposed in plaintext. Pulumi's providers, such as azure-native and Terraform-based providers like Datadog, automatically identify and protect secret outputs, while users can also mark specific outputs and configuration values as secret. Though Pulumi’s built-in secret storage is sufficient for smaller organizations, it integrates with centralized secret management systems like AWS Key Management System, Azure Key Vault, Google Cloud Key Management, or Hashicorp Vault for larger entities. Pulumi simplifies the process of secrets management, automatically protecting secrets during the setup and operation of its services.

Containers are integral to modern application deployment, whether utilizing Kubernetes or serverless architectures, with Docker Hub commonly used for storing images. However, using a personal registry can provide more control and security, as Docker Hub may not always yield consistent images and could expose vulnerabilities. The text explores setting up container registries across major cloud providers—AWS, Azure, and Google Cloud—detailing the process of creating a repository, building Docker images, and pushing them to these registries. Each cloud provider offers unique services but supports Docker or OCI-compliant images, with examples demonstrating the configuration of repositories, implementation of security policies, and management of image lifecycles. The guide emphasizes the importance of securing container images and managing cloud resources effectively using tools like Pulumi, highlighting the need for familiarity with each provider's specific offerings and configuration methods.

Pulumi's Automation API offers a powerful programmatic layer that integrates with Pulumi’s infrastructure engine to facilitate Infrastructure as Software, allowing developers to automate infrastructure provisioning processes using familiar programming languages and tools. Now with Python support, the API can be embedded in software projects to manage infrastructure updates, configurations, and other operations programmatically, utilizing a gRPC interface for executing commands. This API is beneficial for complex workflow orchestrations and platform APIs, as it enables the creation of infrastructure components like AWS RDS databases or S3-backed websites through code, enhancing observability and reliability. Pulumi's Automation API supports Python, TypeScript/JavaScript, and Go, with C# support in development, and can interact smoothly with existing monitoring tools such as Honeycomb, Prometheus, and Datadog, providing a unified approach to managing infrastructure within software applications.

Pulumi has expanded its EKS package, initially available only in TypeScript, to support all four Pulumi languages: TypeScript, Python, .NET, and Go, allowing users to manage EKS clusters more flexibly. This enhancement is part of Pulumi's broader initiative to introduce "multi-language components," which enable components written in one language to be automatically compiled for use in all supported languages, thereby allowing greater accessibility and collaboration across different programming environments. This development will enable infrastructure teams to write components in their preferred languages and share them seamlessly with other teams who may use different languages, fostering a more inclusive and flexible approach to infrastructure as code. The next phase involves creating a seamless authoring experience for multi-language components, which will soon be available for public preview, promising to enhance the ease with which developers can write, build, and publish these components across the global Pulumi community.

Setting up a development environment for cloud engineering involves preparing tools and configurations across different operating systems, cloud providers, and programming languages. This comprehensive guide outlines the process of equipping your environment with essential components such as package managers, cloud provider accounts, and code editors. It emphasizes the use of package managers like Homebrew for macOS, Chocolatey for Windows, and native Linux package managers for installing necessary software including cloud provider CLIs and programming languages such as Node.js, Python, Golang, and .NET. The guide also covers setting up and configuring cloud provider accounts for AWS, Azure, and Google Cloud, highlighting the importance of credential management and authentication using tools like Pulumi. Additionally, it suggests using modern code editors such as Visual Studio Code or Sublime Text for enhanced productivity and provides installation instructions for Pulumi on various platforms. Once the environment is set up, users are encouraged to begin with Pulumi's Getting Started guide and explore example projects to deploy cloud infrastructure, leveraging the tools and configurations established in this setup process.