June 2020 Summaries

The article is the second in a series exploring the use of infrastructure as code to deploy applications with Kubernetes, specifically focusing on deploying applications using Kubernetes objects like Pods, Services, Volumes, Namespaces, and Controllers. It explains how Kubernetes is more than just a container orchestrator, emphasizing its capabilities for scaling applications and maintaining high availability by managing multiple instances of application containers. The article details the lifecycle and states of Pods, the role of Services in routing traffic to Pods, and the importance of PersistentVolumes for data persistence beyond container lifecycles. It also elaborates on the use of Namespaces for organizing virtual clusters and Controllers for managing application states. The article culminates with a practical example of deploying an Nginx application using Pulumi, illustrating how programming languages can offer advantages over traditional YAML manifests by providing enhanced development tools, such as variable definitions, error checking, and code completion, which can reduce errors typically associated with manually editing YAML files. The article hints at future content focusing on more complex application deployments and the use of Helm charts in Kubernetes environments.

Pulumi, a versatile infrastructure as code tool, faces engineering challenges due to its support for three operating systems, multiple programming languages, and nearly 40 providers. The significant size of its Docker container, which includes a comprehensive set of dependencies to accommodate various user needs, has prompted feedback for more streamlined solutions. In response, Pulumi introduced new, language-specific Docker images with version 2.4.0, which significantly reduce the image size by focusing on individual programming languages and their essential dependencies. This change offers a more flexible and customizable experience for users, allowing for quicker pull times and smaller surface areas while maintaining the capability to run Pulumi programs efficiently. These images are available on DockerHub, and although Alpine Linux is not supported due to compatibility issues, users can build their own Alpine-based images using resources provided in the Pulumi GitHub repository.

The text introduces the concept of using infrastructure as code to manage Kubernetes clusters and applications across different cloud providers, including AWS, Azure, and GCP. It highlights Kubernetes as a widely-used open-source orchestration platform that leverages declarative configurations, often defined in YAML files, to manage and deploy containerized applications at scale. The article outlines the process of setting up a Kubernetes cluster using Pulumi, a tool that facilitates infrastructure as code, to create and manage cloud resources. Various configuration parameters such as node type, number of nodes, and passwords are defined to instantiate the clusters, followed by exporting a kubeconfig file for cluster management via kubectl. The article is part of a series that will further explore deploying Kubernetes applications using infrastructure as code, with future installments set to delve into basic Kubernetes objects and higher-level abstractions like deployments and replicasets.

Spinnaker is an open-source, multi-cloud continuous delivery platform that enables rapid and confident software release processes, making it highly compatible with Pulumi for extending its continuous delivery capabilities across various providers. A developer guide has been created to facilitate the installation and immediate use of a Pulumi plugin within Spinnaker, allowing users to create pipelines with a Pulumi stage that displays Pulumi CLI logs in the pipeline execution details. Users are encouraged to provide feedback or ask questions about the plugin through the repository or the Community Slack channel.

AWS Lambda now supports integration with Amazon Elastic File System (EFS), enabling persistent file storage beyond the default 512MB /tmp disk space. Initially, EFS was only available for Amazon EC2 instances, but AWS has expanded its compatibility to various compute offerings like EKS, ECS, and Fargate, and now Lambda. This integration allows serverless functions to easily access network file systems, facilitating scenarios where more disk space is required or where persistent storage needs to be shared across multiple functions or compute services. Using Pulumi, developers can provision the necessary infrastructure to integrate EFS with Lambda, including setting up a Virtual Private Cloud (VPC), EFS file systems, and mount targets. The post demonstrates building an application that utilizes API Gateway to interact with the EFS file system via Lambda functions, enabling file operations like GET and POST without needing the AWS SDK. Furthermore, it explains how to share the EFS file system with other compute services, such as running an Nginx server on Fargate that serves content directly from EFS. This development enhances data management capabilities within serverless environments and offers seamless data sharing with traditional compute resources.

Policy as Code in Pulumi 2.0 enables developers and operators to automate deployment, test resources, and maintain version control using Python, making it a versatile tool for various applications including machine learning, web development, and DevOps. This approach offers significant benefits such as cost savings, enhanced compliance, and efficient deployment by allowing organizations to enforce specific resource criteria through policies, which can be either advisory or mandatory. Pulumi supports two types of policies: ResourceValidationPolicy, which checks individual resources before they are created or modified, and StackValidationPolicy, which assesses all resources in a stack after deployment. These policies are validated during previews and updates to prevent non-compliant resources from being deployed. Examples provided demonstrate the use of Python to create validation functions that enforce policies such as prohibiting public-read access on AWS S3 buckets or Azure Storage Containers, and ensuring Kubernetes services are not exposed as public LoadBalancers, showcasing how Policy as Code can build secure and efficient cloud infrastructures.

Unit testing is crucial for ensuring that deployed cloud infrastructure aligns with expectations, such as verifying that resources' underlying assets have specific properties. Pulumi's unit testing framework, along with language-specific tools like introspection or filesystem calls, can be utilized to implement these tests effectively. Examples include checking that a cloud function's handler is executable, ensuring that a file's mime type matches the specified content type, and verifying that secrets are correctly marked and outputted as secret. These tests, which can be implemented in languages like Python, Node, .NET, and Go, help confirm the correctness of infrastructure components, such as AWS Lambda functions and S3 bucket objects, by validating executable functions, file types, and secret management. The use of Pulumi's testing capabilities allows developers to ensure that their cloud infrastructure is correctly configured and secure.

Jamstack architecture, which stands for JavaScript, APIs, and Markup, offers a modern approach to building websites by deploying them on a CDN and storing content with cloud service providers. This approach provides advantages such as fast deployment of static content, content management with git, automated builds, atomic deploys, and instant cache validation. Despite the simplification of content creation through modern build tools, setting up the necessary infrastructure remains complex, often requiring the use of cloud provider interfaces, CLI tools, or infrastructure as code solutions like Pulumi. The article provides a detailed guide on creating and deploying a Jamstack website using Hugo as a static site generator, Docker for containerization, and AWS services such as S3, CloudFront, and Route 53 for hosting and distribution. Pulumi is utilized to automate the infrastructure setup with either TypeScript or Python, demonstrating how to configure and deploy cloud resources efficiently, while also highlighting the benefits of separating content generation from cloud infrastructure management.

Pulumi's 2.0 release introduces enhanced support for Node.js, Python, .NET, and Go, along with significant improvements to Kubernetes and other cloud resource providers. The update includes refined management capabilities for Kubernetes clusters and application workloads, offering users multiple ways to provision and manage workloads through YAML manifests, Helm charts, operators, and direct API access. The pulumi/kubernetes package now supports Go and .NET, alongside enhanced features like the latest Kubernetes v1.18 release and warnings for deprecated apiVersions. Pulumi expands its support for managed Kubernetes clusters, including additional security group options and envelope-encrypted secrets. The update also facilitates alternative authentication methods, and the generation of kubeconfig files for specific IAM users or roles. Pulumi encourages user feedback to further enhance the Kubernetes experience and offers various resources like PulumiTV and Community Slack for learning and engagement.

Pulumi offers a modern approach to building and deploying applications on Kubernetes by providing developers and operators with tools that enhance the management of Kubernetes resources. Pulumi's Kubernetes SDKs, which are generated from the OpenAPI spec, ensure a 1:1 match with the upstream API across multiple programming languages, allowing developers to utilize the full Kubernetes API surface. It addresses common issues with YAML, such as the lack of conditional logic and strong typing, making it easier to manage resources with built-in error checking and readiness logic. Pulumi also warns users about deprecated API versions and facilitates seamless integration into CI systems. For operators, Pulumi supports the management of Kubernetes clusters, the reuse of existing YAML manifests and Helm charts, and it offers the ability to export resource definitions to YAML for portability. The platform's extension libraries, like kubernetesx, abstract boilerplate code, while still providing full API access. Pulumi enhances collaboration between developers and operators, catering to diverse infrastructure needs across various cloud providers, and provides resources like tutorials and examples to ease the Kubernetes adoption process.

Amazon EKS clusters can utilize IAM roles and policies for Pods to establish precise access control to AWS services by integrating AWS IAM entities with Kubernetes RBAC, thus facilitating least-privileged access for applications. This is achieved through the use of an OpenID Connect (OIDC) provider that verifies Pod identity and collaborates with AWS Secure Token Service (STS) to grant temporary credentials for IAM roles. The process involves creating an OIDC provider within the EKS cluster, configuring IAM roles and policies for specific applications like an S3 app, and deploying the app with IAM-backed Service Accounts. The integration automates Pod identity management and credential injection, enabling secure and effective access to AWS resources. This method can be extended with Kubernetes RBAC for more granular control over API resources, offering a native and secure experience for users. Additional resources and examples are provided for further exploration and implementation of EKS and OIDC provider for Pod IAM.