May 2021 Summaries

The Cloud Development Kit for Terraform 0.4 introduces experimental Go language support, asset construct, and improved Terraform Cloud integration, allowing users to write Terraform configurations in multiple languages, simplify workflows that require assets to be deployed, and streamline output from Terraform Cloud. The new release adds support for the Go language, simplifies deploying files or folders alongside newly provisioned resources, and improves the ability to work with Terraform modules and remote templates. With these enhancements, users can experiment with writing infrastructure as code in Go, deploy full Terraform modules sourced from the same sources as Terraform Core, and divide their applications into logical stacks representing tiers, stages, and regions. The CDK for Terraform project has grown significantly since its 0.1 launch, and users are encouraged to provide feedback on the new experimental Go support through community office hours, discussion forums, and issue trackers.

AWS users now have the option to deploy HashiCorp Consul as their service mesh for Amazon Elastic Container Service (ECS) deployments. This integration allows users to select Consul as their service mesh for ECS deployments, enabling them to take advantage of various service mesh features such as zero trust networking, observability, and reliability. The tech preview of this integration enables non-production deployments on the Fargate launch type, while more secure, production-ready deployments are planned for general availability. Users can deploy Consul using HashiCorp Terraform modules, which automatically add in additional containers to create a full service mesh architecture. This integration expands HashiCorp's ability to support any application on any runtime and is available now with further enhancements planned as it moves towards general availability.

HashiConf Europe 2021 will take place online from June 8-11 and features over 25 technical sessions on three tracks, 10 product workshops, Lightning Talks, and opportunities to connect with HashiCorp engineers and experts via chat and live Q&As. The conference includes highlights such as opening keynotes by HashiCorp Co-Founders Mitchell Hashimoto and Armon Dadgar, infrastructure keynote sessions covering Terraform and Consul, security keynote sessions on Vault and Boundary, application keynote sessions on Nomad and Waypoint, and other sessions including product workshops, In Conversation With Vicki Cheung, and pre-released product videos on demand. Sessions will be recorded and available to view on demand immediately after airing, allowing attendees to catch up at their convenience.

The Terraform AzureAD provider now supports Microsoft Graph API, offering faster response times and more up-to-date information from Azure Active Directory, reducing consistency-related errors during Terraform apply operations. The new support is available in version 1.5.0 of the provider, which also maintains compatibility with most existing Terraform configurations. To use this support, practitioners need to install Terraform v0.12 or greater and version 1.5.0 or later of the AzureAD provider, and can enable it by setting a feature flag in their provider configuration block or an environment variable. The provider will be upgraded to version 2.0, which will exclusively use Microsoft Graph and introduce new features such as better integration with the Microsoft Identity Platform and premium features like Conditional Access and Entitlement Management. Practitioners are encouraged to try out the support and provide feedback on any issues encountered.

HashiCorp Waypoint is an open-source tool that facilitates GitOps workflows, allowing developers to automate and deploy infrastructure using a declarative configuration stored in a Git repository. With the introduction of runners, Waypoint can perform builds, deployments, poll for Git repository changes, and enable deployments on any platform. The tool provides a single runner installation during the Waypoint install step, which can be configured via the CLI or web UI to track Git commits, configure authentication credentials, and store sensitive information securely. Additionally, Waypoint supports extensible configuration options, such as variables within the `waypoint.hcl` file, health check report forwarding, and mutable deployment patterns with Helm and Nomad job files. The project is actively maintained and continues to evolve with new features and updates, including support for variables in the `waypoint.hcl` file and significant foundational feature additions in the next major release.

Boundary is an access management tool that provides secure management for dynamic infrastructure by providing easy access to all hosts and services through a single workflow based on trusted identity. It offers granular, role-based access controls for any networked resource, allowing users to create custom workflows to extend or replace traditional VPNs or firewalls. The tool has recently released new features such as integration with Open-ID Connect providers like Auth0 and Okta, worker filters that enable session routing, and a desktop app for macOS. Boundary provides a powerful way to manage access to hosts and services in a secure and scalable manner.

The Terraform AWS provider now supports predictive scaling policy, a machine learning-based scaling mechanism for Amazon EC2 Auto Scaling, allowing proactive capacity adjustments based on historical demand and usage patterns. This feature uses two metrics, Load Metric and Scaling Metric, to forecast demand and ensure enough instances are launched to meet predicted capacity requirements. Predictive scaling is designed to work in conjunction with dynamic scaling, setting a baseline capacity according to historic patterns while making upward adjustments for deviations from forecasted demand. To use this feature, users need Terraform v0.12 or greater and the latest version of the Terraform AWS provider, and can configure it by employing a new resource `aws_predictive_scaling_configuration` and modifying existing resources such as `aws_autoscaling_policy`.

The Terraform AWS provider now supports AWS AppRunner, a purpose-built container application service that enables customers to build and run containerized web applications with no infrastructure overhead. With App Runner, users can provide their source code or image repository and have it automatically built, deployed, and scaled based on incoming request traffic. To set up App Runner in the Terraform AWS provider, users need to create a few new resources, including an `aws_apprunner_connection`, `aws_apprunner_service`, and `aws_apprunner_custom_domain_association`. Once configured, App Runner provides automatic deployments, high performance, scalability, and security. Additional information can be found in the AWS App Runner documentation and blog post.

HashiCorp has released Nomad 1.1, a new version of its simple and flexible orchestrator for deploying and managing containers and non-containerized applications across on-premises and cloud environments. The release delivers more than 10 new features to improve cluster efficiency, application performance, and volume management, including enhancements to the UI and API, license autoloading, memory oversubscription, reserved CPU cores, and autoscaling improvements. Additionally, Nomad 1.1 supports Consul namespace support, CSI enhancements, and remote task drivers, providing a simplified operator experience and enabling more flexible scheduling. The release is widely adopted by organizations such as Cloudflare, Roblox, Q2, Pandora, and GitHub.

The Cloud Development Kit (CDK) for Terraform is a tool used to define infrastructure using programming languages such as TypeScript, Python, Java, and C#. It allows developers to provision with Terraform using a programming language they're more familiar with. The CDK for Terraform matters because it provides a powerful way to manage entire coding workflows in one language. The tutorial uses TypeScript as the infrastructure code and provisioning language, deploying resources to Microsoft Azure. It covers installing the CDK, setting up the folder structure, configuring providers, writing infrastructure code, running the CDK, and additional resources.

The Terraform AWS Provider now allows users to define default tags at the provider level, simplifying tag management and standardizing infrastructure configuration in accordance with best practices. This feature is available as of v3.38.0 of the Terraform AWS provider, requiring Terraform 0.12 or later and the provider "aws" block with a `default_tags` sub-block to set default tags for all resources except Auto Scaling Groups. The use of default tags simplifies tag management throughout the configuration, making it easier to maintain readable infrastructure as code. Additionally, this feature is designed to support AWS's recommended best practices, allowing users to standardize their Terraform configurations and simplify tag management in their AWS infrastructure.

Docker Compose provides a quick way to explore HashiCorp Consul's core concepts such as service discovery and secure datacenter deployment on local systems. It allows users to deploy a secure Consul datacenter and understand the Consul architecture and security model through tutorials like Deploy a Secure Datacenter with Docker Compose. The tool also offers several quick-start Consul scenarios that can be spun up with a single command, speeding up time-to-concept understanding and reducing local testing complexity. Docker Compose is an implementation of the infrastructure as code philosophy, providing a lightweight configuration file-based approach to defining environments and services. It's commonly used in development environments but not recommended for production deployments, which are better suited for technologies like Terraform.

The Nomad Autoscaler has introduced new plugins and capabilities to simplify consistent autoscaling across cloud providers, enabling automated application and cluster scaling of Nomad workloads and infrastructure. The new plugins include APM, strategy, and target plugins that expand the use cases for autoscaling, allowing for more precise control over scaling policies. The Autoscaler also includes a fixed-value strategy, which provides a simple mechanism to define a baseline for infrastructure, and a pass-through strategy, which allows for external calculations of scaling actions. Additionally, the Autoscaler introduces a threshold strategy, which defines tiers with upper and lower bound values and takes action based on metric crossings. Furthermore, the Autoscaler now offers better cluster scale-in control through node selector strategies, allowing for more flexible client selection mechanisms.

The Cloud Native Computing Foundation (CNCF) has released its latest edition of the End User Technology Radar focusing on secrets management, revealing HashiCorp Vault as the clear winner with the broadest adoption across many companies and industries. Despite being a complex tool, Vault's broad adoption makes sense due to smaller organizations preferring to outsource their secrets management rather than creating an in-house solution. The report concludes that commercial tools like Vault have a higher adoption rate since they remove the complexity of creating an in-house tool, addressing secrets management in a cloud-agnostic way and providing a convenient solution for multi-cloud environments. Other solutions, such as AWS Secrets Manager and Azure Key Vault, were not included due to being fragmented across various levels of maturity and complexity, except for Certificate Manager, which saw high adoption among Kubernetes users. The CNCF End User Technology Radar is a guide for evaluating cloud native technologies, providing insights into the latest developments in secrets management.

The Kubernetes project has adopted the Container Storage Interface (CSI) standard for exposing storage systems to containerized workloads. The Secrets Store CSI Driver is a plugable CSI driver that depends on an additional provider to be defined, with supported providers including Microsoft Azure, Google Cloud, and HashiCorp Vault. The Vault CSI provider is an alternative to the sidecar injector method for retrieving secrets from Vault without native Vault logic built-in. The CSI method simplifies this architecture by not requiring any sidecar containers, rendering secrets before the pod starts, and providing a method to sync secrets into environment variables. However, it uses hostPath volumes to communicate with the CSI driver, which may be disabled in some Kubernetes distributions due to security concerns. The Vault CSI provider has received improvements around stability, support for all secret engines, and authentication with a requesting pod's service account. To implement the Vault CSI provider, users must enable the Kubernetes auth method, configure the Vault policy, create the SecretProviderClass, and define a basic NGINX pod using the CSI driver and SecretProviderClass. The CSI method provides more granular flexibility with access controls and grants the ability to practice principles of least privilege between applications.

HashiCorp Nomad 1.1 introduces several resource control features to improve efficiency and performance, including memory oversubscription, reserved CPU cores, and improved UI visibility into resource utilization. Memory oversubscription allows job authors to set two separate memory limits: a reserve limit for typical usage and a max limit for maximum usage, minimizing unused cluster memory. Reserved CPU cores enable latency-sensitive workloads to reserve entire CPU cores, improving performance and reducing sharing issues. The Nomad 1.1 UI features updates, including exposure of reserved resources and per-task breakdowns on allocation utilization charts, helping operators better understand resource utilization and optimize their clusters.

Controlled Remote State Access` in `Terraform Cloud` and `Terraform Enterprise` adds a new way to establish granular controls for state accessibility, allowing users to manage which workspaces have access to each other's state files while using the `terraform_remote_state data source`. This feature is designed to help organizations improve security by assessing who has access to what information at every step of their infrastructure provisioning workflow. The default sharing configuration for all new workspaces restricts sharing its state with any other workspaces, but existing workspaces will continue to be globally shared within their organization. Terraform Enterprise customers also find a new setting in the admin portal to configure the default setting for every workspace in that environment to either be shared globally or not shared at all. This feature builds on HashiCorp's security-first posture by adding an additional layer of control and protection for sensitive state files, which can contain information such as passwords or private keys.

HashiCorp is sponsoring KubeCon + CloudNativeCon Europe Virtual, hosting informative sessions and answering questions at its digital booth. The company continues to develop Kubernetes integrations with its Terraform, Vault, and Consul products, solidifying its commitment to the CNCF. HashiCorp has recently released Consul 1.10, a service mesh that offers improved features for both Kubernetes and non-Kubernetes users, as well as Nomad 1.1, which enhances efficiency and performance with new resource control mechanisms and CSI volume management capabilities. The company will also be hosting community office hours and product demos during the conference, providing opportunities for attendees to ask questions and learn more about HashiCorp's products.

HashiCorp has released Nomad 1.1, a new version of their simple and flexible orchestrator used to deploy and manage containers and non-containerized applications across on-premises and cloud environments. The beta release delivers over 10 new features that enable more flexible scheduling and simplify the operator experience. These features include memory oversubscription, reserved CPU cores, UI improvements, CSI enhancements, readiness checks, remote task drivers, Consul namespace support, license autoloading, and autoscaling improvements. Nomad 1.1 is a significant update to the platform, introducing new capabilities for managing stateful workloads, improving cluster efficiency, and enhancing the operator experience. The release is currently available as a beta build, but users are encouraged to experiment with the new features in a non-production environment before deploying them in production.