September 2021 Summaries

The HashiCorp Terraform AWS Cloud Control Provider is a new provider designed to bring Amazon Web Services (AWS) resources to Terraform users faster. It aims to increase resource coverage and reduce the time it takes to support new capabilities, making it easier for users to manage their cloud infrastructure in a consistent manner. The provider supports hundreds of AWS resources and can automatically generate codebase for new services, allowing maintainers to focus on user experience upgrades and performance improvements. To use the provider, Terraform 1.0 or later is required, along with an active AWS account in any commercial region excluding China. Users can configure the provider by employing configuration blocks, authenticate using various methods, and create AppFlow flows to integrate with S3 buckets. The provider is still in tech preview, but it offers a promising solution for Terraform users managing infrastructure on AWS.

The HashiCorp State of Cloud Strategy survey reveals key insights into cloud adoption and multi-cloud strategies, with over three-quarters of respondents already using multiple clouds and 86% expecting to use them in two years. Digital transformation was a top factor driving multi-cloud adoption, but avoiding vendor lock-in and cost reduction were also significant factors. Cloud investment continues to grow, with 15% of respondents budgeting at least $10 million on their initiatives, but managing spend is a challenge, with over 40% overspending. The cloud skills shortage is a major issue, with 57% of respondents citing it as a top-three multi-cloud challenge. Finally, the COVID-19 pandemic has accelerated cloud adoption, with most respondents calling its impact low or moderate, although cloud's ability to adapt to changing conditions remains valuable.

HashiCorp has made its Consul service mesh available on Amazon Elastic Container Service (ECS) as a generally available product. The service mesh provides secure deployment, support for various launch types, and self-managed deployment options. AWS users can now select Consul as their service mesh for secure ECS deployments. The new features include gossip encryption, Transport Layer Security (TLS), and access control lists (ACLs) for secure communication between agents and servers. HashiCorp provides a Terraform module to deploy Consul on ECS and offers tutorials and documentation to help users get started with the service mesh.

The HashiCorp State of Cloud Strategy Survey reveals that infrastructure as code and provisioning tools are at the forefront of the movement to multi-cloud, with 90% of organizations finding automation tools critical to overcoming cloud challenges. The survey found that provisioning was the most widely used category of infrastructure automation tooling, with 78% of European/Middle East/Africa respondents using automated provisioning tools, compared to 65% of those with annual cloud budgets under $100,000. The COVID-19 pandemic accelerated digital transformation initiatives, with 54% of organizations reporting an increase in multi-cloud adoption. However, the top challenge hindering organizations' ability to operationalize multi-cloud was a shortage of key skills, cited by 57% of respondents. Open source provisioning tools are popular, with two-thirds of respondents using or planning to use some form of open source automation tooling for provisioning. The survey also found that adoption of SaaS solutions would seem to be a natural response to skills shortages and complex manual processes.

HashiConf Global` is an upcoming virtual conference by `HashiCorp`, scheduled for Tuesday–Thursday, October 19-21, 2021. The event will feature more than 45 technical content sessions on three stages, keynotes from `HashiCorp Co-Founders Mitchell Hashimoto and Armon Dadgar`, live Q&As with multiple speakers, hands-on labs led by `HashiCorp` experts, the `HashiCorp Zone` for product demos and interactions, a `HashCorp User Group contest` to thank participants, and rebroadcasts for viewers in the Asia Pacific region. The conference platform is already open, allowing attendees to save sessions, sign up for lightning talks, and vote for their local `HashiCorp User Group`.

In this post, we'll walk through setting up and using the Amazon Virtual Private Cloud (VPC) for Terraform on AWS module. To deploy this module, you need to install Terraform, sign up for Terraform Cloud, configure your API access, generate a token, add it to the CLI, and then deploy your first module. The VPC pattern allows you to set up a VPC that provides a networking foundation based on AWS best practices for your infrastructure. You can use this module to quickly deploy Terraform-based deployments on AWS by calling it multiple times within the same solution or in separate ones. There are currently 15 modules available, including the VPC environment, which has both public and private subnets where you can launch AWS services and other resources based on best practices. Each module includes availability from registry, getting started provisioning instructions, and a readme that details how to import it into your Terraform files. After setting up the basics, you can deploy this module by cloning the repository, building the workspace in Terraform Cloud, creating a remote file, and then applying the configuration. The module uses Terraform Cloud, which is free for up to five users, and provides a developer preview of Terraform modules on AWS, published under an open source license with the source code available on GitHub.

The inaugural HashiCorp State of Cloud Strategy survey highlights the complexity and importance of security issues in cloud adoption, with 47% of respondents citing security concerns as their second biggest cloud inhibitor. Security was a top concern for financial services organizations and healthcare/biotech companies, but less so for entertainment/media firms and software/services companies. Geographically, security concerns were most pervasive in the Asia-Pacific region, while respondents in Europe/Middle East/Africa had the lowest percentage. Small businesses were least concerned about security, with 38% of these respondents calling it a top-three cloud inhibitor, compared to 55% of large enterprises. Security was also cited as a multi-cloud driver for 16% of respondents, particularly in the public sector and financial industry. Data/privacy protection was the top cloud security concern, citing staff and skills shortages as a major challenge. Almost 90% of respondents agreed that right tools are essential for managing multi-cloud environments, with infrastructure automation tools being critical for security, but only half of respondents currently using them in this area.

This article discusses the pros and cons of managing credentials and secrets in Terraform Cloud and Enterprise, highlighting five different approaches using Terraform workspaces, Terraform Agents, HashiCorp Vault, direct integration with a Vault plugin, and CI/CD integrations. It emphasizes the importance of considering security principles, such as uniqueness, ease of rotation, dynamism, and protection with Role-Based Access Control (RBAC). The article also notes that each approach has its strengths and weaknesses, and that there is no one-size-fits-all solution for managing credentials and secrets in Terraform Cloud and Enterprise.

The latest release of Cloud Development Kit (CDK) for HashiCorp Terraform 0.6 introduces several key improvements, including Jest-based unit testing via snapshot tests and assertions, improved handling of native Terraform functions, Bash and zsh shell completion, and various bug fixes. The new testing workflow provides a high level of test coverage with a few test cases, allowing developers to write infrastructure as code and ensure the quality of their generated configurations. Additionally, CDK for Terraform 0.6 adds support for testing in other languages, including Python, C#, Java (with Go experimental), and will include more language-specific testing in future releases. The release also includes a comprehensive list of enhancements and bug fixes, as well as a variety of other improvements, and is accompanied by tutorials, guides, and documentation to help developers get started with the new testing functionality.

Terraform Enterprise has introduced several new features to enhance operational visibility and accessibility for businesses, including a workspace overview page, structured run output, log forwarding capabilities, and support for Terraform Agents, which allow workflows to be performed in private or segmented networks with consistent control plane management. The new features aim to address the growing need for organizations to understand what's happening in their infrastructure and when, providing immediate access to relevant information and increasing the ability to run workflows anywhere. By adding a more visually appealing representation of output during Terraform workflows and enabling log forwarding capabilities, Terraform Enterprise enhances contextual awareness and observability, making it easier for operators to tell exactly which resources have been modified and in what manner.

HashiCorp has released version 0.6 of its Boundary identity-based secure remote access solution, which adds support for Linux with Boundary Desktop 1.3.0, permissions enforcement improvements in the admin console, and Terraform provider support for managed group configuration. The new release introduces features such as dynamic UI tailoring to individual users' permissions, automated toggling of UI features based on user permissions, and improved security controls. Boundary Desktop now offers feature parity with its existing macOS and Windows clients, allowing Linux users to easily manage remote access sessions. The Terraform provider support enables managed group configurations, which automate the population of Boundary groups based on permission claims maintained by an external identity provider.

The Microsoft Cloud Adoption Framework (CAF) provides guidance and best practices to adopt Microsoft Azure, offering two paths to adoption: enterprise scale and CAF Terraform modules. Enterprise scale tackles the big picture, while CAF modules take a smaller approach to build out elements that can be customized to deliver an outcome from different paths. The CAF module allows users to create resources on Azure and provision defined resources in an Azure subscription, using reusable code to pre-provision workloads through code. Terraform Cloud is used as a tool for deploying the CAF modules, and it supports version control systems such as GitHub. Users can choose between enterprise scale deployment of controls, guardrails, and management groups, and use the CAF module to deploy workloads at high speed.

HashiCorp has announced that its Terraform Cloud run tasks feature is now available in beta access, allowing organizations to integrate third-party tools into specific points during the Terraform Cloud workflow. This feature was previewed earlier this year and is now open to all existing Business tier customers. Run tasks enable integration with external systems for security checks, cost estimation, code scanning, and other purposes, receiving a passed or failed response back to Terraform Cloud based on the task's enforcement setting. Several technology partners have pre-built integrations available, including Bridgecrew, cloudtamer.io, Infracost, Lightlytics, Refactr, and Snyk. The feature is open for anyone to develop against without formal approval, with documentation available for building custom integrations. Terraform Cloud remains free to try, and organizations can upgrade to the Business tier at any time.

The auto-config feature in Consul is a highly scalable method to distribute secure properties and configuration settings to all Consul agents in a datacenter, reducing the technical overhead associated with securing an environment. Consul clients configured with auto-config use JSON web tokens (JWTs) to securely retrieve gossip encryption keys, TLS certificates, ACL settings, and other configuration properties from Consul servers, which are generated and validated using production-grade tools such as HashiCorp Vault. The auto-config workflow involves configuring the Consul server cluster as a JWT authorizer, generating a JWT with claims data, setting it in the Consul client auto-config configuration file, and submitting an auto-config request to the Consul server cluster for validation, which then sends security settings to the Consul client, merging them into existing client settings. To get started, check out the Automate Consul Agent Security with auto-config tutorial or explore other resources such as the Consul Reference Architecture and Docker documentation.

HashiCorp has added a note to its Terraform contribution guidelines to provide clarity on the process for community members and commercial customers. The company appreciates any contributions from its community, which have become integral parts of Terraform. Despite incredible growth, HashiCorp is facing scalability challenges as it manages an increasing volume of contributions. To address this, the company has updated its CONTRIBUTING.md file to set expectations for potential contributors about review timelines, and it expects to return to normal processes in a few weeks. HashiCorp remains committed to Terraform and encourages community members to contribute and join the team if they want to help scale the project.

The HashiCorp State of Cloud Strategy Survey reveals that software companies and telcos are leading the way in getting value from cloud and multi-cloud, while the public sector is lagging behind. The survey found that staffing and skills shortfalls are a universal concern across industries, but unique business-based dynamics affect cloud provider choices and budgets. Telcos and software companies are more likely to reap benefits from multi-cloud success, while the public sector struggles to realize similar benefits. Cost is a top cloud concern for telcos, software, and media companies, with security being a major issue in financial services. The survey also found that many telcos and retailers shrugged off COVID-19's impact on their cloud plans, and that infrastructure automation tools are crucial for managing multi-cloud environments. Overall, the survey suggests that industries that have been focused on digital transformation for longer are seeing more benefits from multi-cloud adoption.

F5 has validated its integration with HCP Vault, allowing users to improve their security posture by using short-lived dynamic SSL certificates, automate application updates, increase collaboration, and deploy on-premises or public cloud. Garantir's GaraSign integrates with HCP Vault's Transit Engine, enabling secure use of cryptographic private keys without exposing them to end-users. GitHub provides two integrations for users to authenticate with HCP Vault using GitHub tokens and retrieve secrets from a GitHub Actions workflow. Okta customers can connect their identity cloud to HCP Vault, while Percona users can store credentials in a central location and generate dynamic users for multiple databases. Rafay Systems has developed an integration to simplify the process of integrating HCP Vault with Kubernetes workloads. Resolve Systems' IT automation workflow engine fetches credentials from HCP Vault for use within automations and runbooks, providing a secure way to manage sensitive data.

HashiCorp Vault offers zero trust security solutions for Microsoft Azure, enabling organizations to build secure cloud environments. The partnership between HashiCorp and Microsoft has led to integrations that make HashiCorp products work seamlessly with Azure's native capabilities. Five common use cases for using Vault with Azure include identity-based authentication to Vault using Azure Active Directory, dynamic secrets generation for Azure resources with Vault, granular authorization to Azure resources with Azure Managed Service Identities, encryption of everything including data with Vault Key Management Secrets Engine (KMSE) for root of trust with Azure Key Vault, and automating secrets management with Vault Agent on AKS. Additionally, best practices for running Vault on Azure include auto-unsealing Vault with Azure Key Vault, using Azure Key Vault to establish secure communication with Vault via TLS, streamlining Vault image creation with Azure Shared Image Gallery, encrypting managed disks on Vault (OS and data) with Azure KEK, and achieving high availability for Vault with Azure Availability Zones.

The HashiCorp State of Cloud Strategy Survey reveals that service mesh is a high priority for many organizations, with 68% of respondents at some level of implementation. However, only 10% have completed their implementations, and most are still in the planning or trial stages. Networking security concerns focus on data protection, data theft, and regulatory compliance. The survey also highlights the importance of addressing staff/skilling issues and manual processes as major obstacles to service mesh adoption. HashiCorp's Consul-Terraform-Sync solution can help bridge these gaps by providing automation and reducing the need for ticketing systems. Additionally, security is a top concern for both service mesh explorers and non-mesh users, with data protection being a key focus area.

Consul-Terraform-Sync (CTS) 0.3 is now generally available for HashiCorp Terraform Enterprise, offering compliance-driven network infrastructure automation capabilities that reduce the burden on operators caused by manual ticketing systems. The new release provides regex support for service triggers, enabling tasks to be triggered only for services matching a specified pattern across the entire Consul catalog. CTS integrates with Terraform Enterprise, allowing users to automate common tasks right away and improving automation controls, reducing risk, simplifying auditing, and applying compliance policies. Several launch partners, including A10 Networks, Check Point Software Technologies, Cisco, Citrix, F5, and Palo Alto Networks, have partnered with HashiCorp to expand the CTS module library and provide additional capabilities. The new release is available for download on the Consul documentation page, and users can get started by referring to the Consul documentation and HashiCorp Learn guides.