March 2020 Summaries

The detailed examination of virtual machines focuses on the process of creating, configuring, and provisioning virtual machines using cloud service providers like AWS, Azure, or Google Cloud Platform. It highlights how to set up an EC2 instance with necessary resources such as network access, security groups, and storage using Pulumi, a modern infrastructure as code tool. The example provided demonstrates configuring security groups for SSH and HTTP/HTTPS access, selecting Amazon Machine Images (AMI), and attaching Elastic Block Storage (EBS). Additionally, the text discusses the use of Pulumi's dynamic providers to provision existing instances by executing custom scripts during deployment, specifically illustrating the installation and configuration of PostgreSQL on an EC2 instance. The process involves copying configuration files via SCP and running commands to set up the database, all orchestrated through a series of dependent commands to ensure the correct deployment sequence. The article concludes with a promise of exploring serverless infrastructure deployment in a subsequent discussion.

Ringo De Smet, founder of Cumundi, highlights the company's use of Pulumi for creating reusable infrastructure code libraries, which are essential for helping businesses efficiently adopt cloud infrastructure. Pulumi facilitates rapid iterations through the build-test-release cycle and supports Cumundi's approach of encapsulating non-functional requirements into code libraries, thereby maintaining clean and modular infrastructure across different cloud platforms. The article demonstrates the implementation of Test-Driven Development (TDD) practices in Pulumi, showcasing how ComponentResource and properties like parent and aliases allow for the effective refactoring of infrastructure code. By using Pulumi, Cumundi caters to customer needs by setting up infrastructure and visualizing resource relationships through a state graph on the Pulumi platform, while also addressing challenges like overcoming code duplication and maintaining clear parent-child resource relationships.

The second part of a series on securely managing AWS credentials in CI/CD systems explores the risks and best practices associated with providing AWS credentials to third-party services and introduces a Pulumi program to automate the rotation of access keys. The article emphasizes caution when sharing AWS credentials due to the potential for exposure, even in secure CI/CD environments, and critiques using secrets managers as they might not enhance security but add complexity. Instead, it advocates for automating the rotation and revocation of AWS credentials to mitigate risks, using a Pulumi-based serverless application, the AWS Credential Rotator 9000, which periodically creates new access keys and updates dependent systems like Travis CI. This approach ensures that credentials remain volatile and reduces the impact of accidental disclosure, while also setting the stage for the next post in the series, which will discuss using IAM Roles for temporary access to additional resources in AWS environments.

Pulumi has announced support for unit testing with Node.js, Python, .NET, and Go in recent releases, allowing developers to test infrastructure resources before deployment using familiar tools and frameworks. This approach enables testing of resource configurations and responses without the need for actual deployment, thereby accelerating infrastructure development and deployment processes. By integrating testing capabilities into Infrastructure as Code, Pulumi aims to provide early bug detection, cleaner code, and easier refactoring. The announcement includes examples of how to set up tests in Python and Go, demonstrating the use of existing test tools, dependency mocking, and the ability to run code without the engine. Pulumi has made these features available ahead of the 2.0 release and encourages feedback from users through its Community Slack channel, with additional resources and documentation to be provided in the future.

The shift from monolithic applications to distributed systems such as serverless, microservices, Kubernetes, and virtual machines highlights the importance of abstraction in building resilient infrastructures. Abstraction allows components to function independently by encapsulating behavior and decoupling code, which is also crucial in infrastructure to declare behavior or state without focusing on implementation details. Virtual machines optimize hardware utilization by sharing host resources, while serverless architecture, often managed by third-party cloud providers, allows developers to choose languages optimized for tasks, supporting both synchronous and asynchronous calls. Kubernetes orchestrates container deployment, scaling, and management, offering features like load balancing and automated rollbacks, whereas microservices promote independent deployment of services encapsulating business capabilities, often adhering to the 12-Factor app principles. Each architecture pattern, including those facilitated by platforms like Pulumi, emphasizes creating reusable components that abstract configuration details to enable a plug-and-play architecture, with further exploration of these technologies planned for future discussions.

Pulumi has implemented several measures to address the challenges posed by the COVID-19 pandemic, focusing on the health of employees and the continuity of business operations. The company swiftly transitioned to a fully remote work model to ensure employee safety and support social distancing, leveraging its pre-existing remote-friendly infrastructure. Despite the pandemic, Pulumi has maintained its service delivery and continues to release new features, while emphasizing enhanced communication and knowledge sharing within the team. The company has shifted to digital channels to engage with customers, launching initiatives such as weekly live streams and online events, including an upcoming major product launch. Pulumi remains committed to supporting its customers, particularly those in government and healthcare, and invites engagement for further collaboration and support.

Pulumi Azure 2.0 introduces a major update to the Pulumi Azure provider, built on the 2.0 release of the upstream provider, enhancing features and implementing several breaking changes. Key improvements include streamlined Azure Storage resources with native static website support, new resources for Virtual Machines and Virtual Machine Scale Sets, and removal of deprecated Azure Active Directory resources, which now have their own provider. The update shifts callback function deployments from ZipBlob to Blob, adjusts default runtime versions for serverless functions, and introduces custom timeouts for lengthy resource creation processes. The release aims to prevent accidental resource adoption by rigorously checking for existing resources and offers guidance for migrating to the new version, with updated templates and community support available for users during the transition.

The guide provides a comprehensive overview of best practices for managing AWS credentials within a CI/CD pipeline, specifically focusing on using Pulumi for Infrastructure as Code. It emphasizes the importance of securely providing AWS credentials to CI/CD systems to automate cloud infrastructure updates without compromising sensitive credentials. The article suggests creating a dedicated IAM User, also known as a "robot account," for CI/CD pipelines and assigning permissions through IAM Groups to enhance security. It introduces the concept of using the AWS API sts:AssumeRole for granting temporary credentials with necessary permissions, rather than directly assigning permissions to the IAM User, thus maintaining a higher level of security. This is the first installment in a series that aims to explore AWS credential management in-depth and its application within CI/CD environments, promising future discussions on credential rotation and security concerns.

In early 2020, significant advancements were made in the Pulumi platform, with major updates to the .NET and Go SDKs, the introduction of the Policy as Code framework CrossGuard, and various additional features aimed at improving user experience and functionality. The .NET SDK received enhancements for better alignment with .NET developers, including new examples and closing gaps in core Pulumi features, while the Go SDK underwent a comprehensive overhaul to incorporate strong typing across all resource APIs, moving away from map-based data structures. CrossGuard, designed for writing and enforcing policies in real code, became ready for testing, offering capabilities like policy packs and server-side enforcement for enterprise users. Additional improvements included stack export features, audit logs, runtime mocking for infrastructure testing, and expanded provider support, along with ongoing efforts to improve API documentation, multi-language library architecture, and performance enhancements for a more efficient CLI experience. Pulumi encouraged user feedback and engagement through community platforms to refine these developments further.

BigData Boutique, led by Founder and CTO Itamar Syn-Hershko, utilizes Pulumi to benchmark Elasticsearch configurations across various cloud providers, enhancing their capacity to deliver cost-effective and high-performance solutions. Pulumi's infrastructure-as-code capabilities allow for parallel testing and rapid iteration of infrastructure configurations, leveraging familiar programming languages to shorten the development cycle. This approach facilitates the automation of complex infrastructure setups, enabling BigData Boutique to optimize Elasticsearch clusters for their clients by identifying the most effective hardware configurations that balance cost and performance. The process involves rigorous benchmarking and analysis, reducing the number of variables to test and using data-driven insights to recommend optimal configurations. These benchmarks are automated to ensure precision and are conducted in parallel across different infrastructures, which are then analyzed to guide configuration adjustments. This systematic approach allows BigData Boutique to provide tailored, efficient solutions for clients, ensuring their Big Data systems are both cost-effective and performant.

Pulumi offers a solution for generating Kubernetes YAML manifests using familiar programming languages, such as TypeScript, JavaScript, Python, and .NET, making it easier to integrate into existing CI/CD workflows. The v1.5.4 release of pulumi-kubernetes introduces the renderYamlToDirectory option, allowing users to render YAML while leveraging the benefits of their chosen programming language, such as using libraries and mixing infrastructure configuration. This feature enables efficient templating, with manifests kept in sync with program changes, and allows for deployment with kubectl. However, users should be aware that YAML-rendered resources are not created on a Kubernetes cluster, meaning certain server-side computed information will be unavailable, and secret values will appear in plaintext, requiring protective measures. Pulumi, open-source and free to use, provides additional resources and community support for users interested in managing infrastructure and Kubernetes through code.

The Pulumi Service has introduced several new features aimed at simplifying stack management, permissions, and organizational tasks. These updates include first-class support for stack tags, allowing users to create, update, and delete tags directly from the console, as well as new search and filtering capabilities based on these tags. The service now offers deep linking into CI/CD systems, enabling users to access their CI/CD tool of choice directly from the Pulumi Service interface. Additionally, improvements have been made in pretty printing JSON, viewing multi-line data, and downloading stack checkpoint files for advanced scenarios. The Enterprise Edition now features a reverse stack permissions view, allowing administrators to review and manage access permissions more effectively. Performance enhancements have been implemented for log rendering, and audit logs are now available to help track changes within organizations. These developments are part of Pulumi's ongoing efforts to enhance its service, with more features planned for the future.