August 2020 Summaries

The blog post provides a comprehensive guide on transitioning a Python AWS application from Flask and Redis to Django and MySQL, emphasizing the similarities in their underlying deployment infrastructure. The tutorial includes detailed instructions on setting up a new directory, initializing a Pulumi project, and creating a Django project, while stressing the importance of security by managing secrets with Pulumi. It describes containerizing the application using Docker, configuring necessary AWS components like ECS Cluster, VPC, and RDS instance, and creating Docker images for deployment. Two independent ECS services are set up: one for configuring the MySQL database and another for running the Django website, highlighting the risk of exposing sensitive credentials if combined. The guide concludes with instructions on exporting the application’s DNS name and accessing the Django administration panel to manage the application, while also indicating future plans to explore other technologies such as PostgreSQL, Express, React, and Node.js. The full code and detailed explanations are available on GitHub.

Pulumi has enhanced the Python programming experience by introducing type annotations to its APIs and allowing the use of strongly typed data classes instead of raw dictionaries. This update improves the editing experience in IDEs such as PyCharm and VS Code and enhances type checking and consistency. Type annotations, based on PEP 484, enable tools like mypy to catch errors earlier, while the use of data classes simplifies the process of setting nested values and ensures a more Pythonic approach with snake_case arguments. These changes also allow for more strongly-typed access to nested outputs, maintaining compatibility with existing programs using dictionaries. Additional improvements include enhanced hover tooltips in IDEs and streamlined completion lists. These updates, available in the latest releases of various cloud providers, aim to significantly improve the development experience for Pulumi users.

Pulumi has introduced native support for integrating GitLab Merge Requests, enhancing the ability to confidently approve merge requests by allowing users to preview and validate infrastructure changes within the GitLab CI/CD workflow. This integration leverages GitOps principles, using Pulumi's commands to preview and update infrastructure changes as part of a streamlined workflow, ensuring that changes are validated before application. Users familiar with both Pulumi and GitLab CI/CD can now centrally manage infrastructure changes during the merge request process, thus improving visibility and reducing the risk of missing important updates. The integration aims to improve application delivery by providing a seamless way to manage infrastructure changes, and users are encouraged to explore detailed setup instructions in the documentation and engage with the Pulumi Community for feedback and suggestions.

The blog post describes the process of creating and initializing a MySQL database using Pulumi, focusing on overcoming the limitation of Pulumi’s MySQL provider, which does not support creating and populating tables. The author extends Pulumi’s functionality by implementing a Dynamic Provider that connects to a MySQL server, initializes tables, and inserts starting data during deployment. The process involves setting configuration variables, writing a Dynamic Provider class to handle database operations, and integrating the provider into a cloud infrastructure setup involving AWS services like VPC, subnets, and RDS instances. This approach enhances security by limiting the sharing of admin credentials and demonstrates Pulumi's extensibility. The post concludes with a mention of future plans to integrate the setup with a Django frontend and provides a GitHub link for the complete code and detailed explanations of each component.

Pinpoint utilizes Kubernetes, enhanced by managed services like AWS EKS, to efficiently deploy and scale applications, while leveraging Pulumi to streamline infrastructure management through open-source libraries that integrate with AWS APIs. Pulumi allows Pinpoint to simplify infrastructure-as-code by providing powerful constructs such as for-loops and strong type-checking, which reduce the need for complex scripts and allow for easy updates. To manage costs, Pinpoint employs EC2 spot instances as worker nodes, using Pulumi's AWS library to dynamically set spot prices based on current on-demand rates. Additionally, Pulumi aids in managing third-party services on Kubernetes clusters, offering flexibility with tools like Helm and Kustomize, and allowing for transformations to adjust manifests, such as addressing namespace issues with the Traefik Helm chart. This approach has enhanced Pinpoint's ability to maintain infrastructure and deployment configurations, paving the way for further adoption of Pulumi as the company grows.

Pulumi has introduced support for the Open Policy Agent (OPA) Rego language in its CrossGuard policy-as-code framework, allowing users to write policies in either JavaScript, TypeScript, Python, or Rego. This integration broadens the spectrum of policy authoring options, enabling users to leverage the declarative Rego language for creating and enforcing policies across Pulumi deployments in various environments, including cloud and Kubernetes. The CrossGuard framework supports policy enforcement during deployment previews, ensuring policy violations are detected early, and offers both advisory and mandatory recommendations. This allows organizations to maintain high standards of compliance, security, and best practices across their infrastructure projects. With the addition of OPA, Pulumi now offers a versatile platform for authoring and enforcing infrastructure policies, catering to diverse infrastructure teams' needs by supporting multiple programming languages. Integration with tools like Visual Studio Code enhances the development experience, making it easier to manage infrastructure and policies within a comprehensive IDE environment.

Vova Ivanov, a Pulumi summer intern, details his experience modernizing a web application using Pulumi by transitioning an existing TypeScript voting app example into Python with Flask for the frontend and Redis for the backend. He outlines the step-by-step process of cloning the repository, setting up a new Pulumi project, and configuring AWS infrastructure, including the creation of an Elastic Container Service (ECS) cluster, Virtual Private Cloud (VPC), security groups, IAM roles, and an Elastic Container Registry (ECR) for Docker images. The blog post demonstrates setting up both the Redis backend and Flask frontend, using AWS services such as ECS, EC2, and ECR to manage containerized applications and their networking. Ivanov emphasizes the ease of standardizing on a single programming language for frontend, backend, and infrastructure using Pulumi, and previews future posts on further modernizing the app by migrating from Flask to Django and Redis to Amazon RDS. The complete code for this project is available on GitHub, providing a comprehensive guide for developers looking to implement similar infrastructure changes.

CustomResource in Kubernetes allows users to enhance the API with their own types by using CustomResourceDefinitions (CRDs) that include an OpenAPI schema, though this extensibility often involves complex YAML definitions. To simplify the management of CustomResources, the new crd2pulumi tool generates types in any Pulumi-supported language, making it easier to handle the intricacies of CRDs like cert-manager or Istio, which can involve extensive and complex YAML schemas. By using programming languages for defining these types, developers benefit from IDE features such as type checking and autocompletion, reducing errors compared to YAML management. Examples like the CronTab and cert-manager demonstrate how crd2pulumi facilitates the creation and management of CustomResources in different programming languages, offering a more efficient and error-free approach. Pulumi provides a robust platform for managing Kubernetes resources, allowing developers to focus on problem-solving rather than dealing with YAML intricacies.

Pulumi has announced several new open-source projects to enhance its Kubernetes support, developed in collaboration with leading cloud-native engineering teams. These new features include the Pulumi Kubernetes Operator, which offers a Kubernetes-native GitOps experience, and the kube2pulumi tool for converting YAML manifests into Pulumi-supported programming languages, thus simplifying Infrastructure-as-Code for complex Kubernetes applications. Additionally, Pulumi now supports Open Policy Agent (OPA)-based policies within its CrossGuard framework, and the crd2pulumi tool allows for the creation of strongly-typed APIs from Kubernetes Custom Resource Definitions, facilitating the management of API extensions. The integration of Helm 3 and Kustomize into Pulumi programs further enriches its Kubernetes capabilities. These innovations aim to streamline infrastructure delivery, enforce policy, and apply robust software engineering practices to Kubernetes workloads, making Pulumi an attractive option for modern cloud projects.

Kube2pulumi is a tool designed to streamline the conversion of Kubernetes manifests from YAML files into code in various programming languages, enhancing productivity by utilizing programming language capabilities such as strong typing, linting, and refactoring. This tool allows users to bypass direct YAML manipulation, converting manifests into code for languages like TypeScript, Python, C#, and Go, demonstrating its effectiveness with examples ranging from simple namespace conversions to more complex scenarios involving multiple resources. By leveraging programming language features, kube2pulumi helps maintain synchronization across resources, enabling developers to manage Kubernetes infrastructure as a software project with ease. Additionally, Pulumi provides further flexibility and power by allowing developers to abstract resource details, configure stack values, and implement custom logic, thus offering significant enhancements to Kubernetes management through code. Pulumi is open-source and encourages users to explore its capabilities for infrastructure management across various platforms, with further resources and community support available for learning and development.

Version 2.0 of the Pulumi Kubernetes Operator introduces a Kubernetes controller that facilitates the deployment of cloud infrastructure within Pulumi Stacks, enhancing collaboration between Kubernetes developers and operators. This operator allows users to manage a wide range of resources, including virtual machines, managed Kubernetes clusters, and serverless functions, by creating Pulumi Stacks as first-class API resources. Users can deploy these stacks on various cloud platforms such as AWS, Google Cloud, and Azure using supported programming languages like TypeScript, Python, .NET, and Go. The operator integrates smoothly with CI/CD processes and supports advanced deployment strategies like Blue/Green, enabling seamless software updates without downtime. Detailed examples, including deploying an NGINX Stack and creating a Kubernetes cluster on AWS EKS, are provided to guide users through the process. For those interested in exploring further, GitHub resources and community engagement opportunities are available.

DevOps emphasizes the collaborative effort between developers and operators to manage applications beyond the production stage, referred to as "Day 2" activities, which include managing, changing, and monitoring cluster components. These tasks involve handling cluster resources, scaling applications, and ensuring system health through logging and metrics analysis. Pulumi provides tools to streamline these processes by using infrastructure as code, allowing for efficient secrets management, cluster updates, and real-time resource monitoring. The framework facilitates seamless application scaling and deployment, exemplified through tutorials that guide users on how to handle practical scenarios such as zero-downtime migrations and performance-based scaling using technologies like Prometheus. Pulumi's support for various logging and monitoring tools further aids in the comprehensive management of Kubernetes clusters, offering detailed guidance through a series of independent articles that build on foundational concepts.

The v2.8.0 release of Pulumi introduces a feature allowing users to change the secrets manager associated with their stacks, enabling them to rotate or migrate secrets providers as needed. This update includes a command to change the secrets provider for an existing stack, ensuring re-encryption of configuration values and migration of secrets within the state file. Additionally, users can now create new stacks based on existing configurations, with options to specify a new secrets provider or copy configurations between stacks programmatically. This functionality streamlines managing infrastructure as code, offering flexibility in handling secrets and stack configurations. Pulumi remains an open-source tool, encouraging community engagement through various platforms.