August 2021 Summaries

The HashiCorp Terraform provider ecosystem has grown rapidly, with over 1,300 verified providers now available, surpassing the 1,000 threshold announced just a couple of months ago. This growth is good news for users hoping to deploy and manage infrastructure across products in multi-cloud environments using infrastructure as code. The ecosystem welcomes 27 new verified providers, including notable companies like 1Password, 3DS Outscale, Backblaze, Cisco ASA, Citrix, CloudScale, CodeFresh, Cohesity, ConsenSys, Datastax, DNSimple, Elastic, Fortinet, Illumio, Infracost, LogicMonitor, Ngrok, Nirmata, Octopus Deploy, Pure Storage, Rafay Systems, Spectro Cloud, Shoreline Software, and Valtix. These providers offer a wide range of services, from cloud infrastructure to security, monitoring, and DevOps tools. The ecosystem also celebrates reaching over 150 verified providers, with many more available through the Terraform Registry.

HashiCorp Vault's AppRole authentication method provides a secure way for applications to authenticate and retrieve secrets. The process involves creating a role and policies for the app, requesting and receiving the Role ID, provisioning the application, generating and delivering the Secret ID, providing the Secret ID wrapping token to the authorized application, unwrapping the wrapped Secret ID, authenticating to Vault using the provided Role ID and Secret ID, and finally using the returned token to access secrets in Vault. The AppRole method is preferred over direct token assignment due to its ability to associate identity with the application and provide a trusted broker for applications easily and effectively. Response wrapping of the Secret ID provides benefits such as concealment, exposure limitation, and tamper-evidence against unauthorized use or interception of credentials. By following best practices, such as using identities tied to minimal-permission policies, providing credential components just-in-time by separate paths, making sure credentials are used immediately, and monitoring and alerting on errors in unwrapping or authentication, the AppRole method can be effectively operationalized for scaling use cases.

The HashiCorp State of Cloud Strategy Survey reveals that infrastructure tool buyers seek support, compliance, and management features, as well as a faster return on investment. Most practitioners prefer to build on open source projects and run the software themselves, but also use cloud-enabled commercial tools for security. The survey found that respondents were more likely to use cloud-enabled commercial tools for security than for other infrastructure components, and nearly four times as many respondents use open source projects for their infrastructure scaffolding. Companies buy infrastructure tools primarily for support, compliance, and management features, with a growing appreciation for agility, distinctive competence, and ROI. The COVID-19 pandemic led to an increase in the use of open source software by 39% of respondents, who turned to it to rapidly respond to new business needs.

The Terraform AzureAD provider version 2.0 exclusively uses the Microsoft Graph API, marking a significant transition from the legacy Azure Active Directory Graph API. This release includes numerous schema and behavioral changes to enhance user experience and improve object management. The new version requires authentication using fine-grained API roles, rather than directory roles, and introduces new required UUID properties for application roles and OAuth 2.0 permission scopes. Additionally, it adds a new resource and data source to manage pre-authorized applications and improve readability of application configurations, respectively. It is recommended to consult the upgrade guide before upgrading, as this release contains several breaking changes, and to use Terraform 1.0 for optimal functionality.

HashiCorp Vault is a comprehensive data protection solution that offers various methods to protect sensitive information, including encryption, data masking, and tokenization. Each method has its tradeoffs and is suited for specific use cases. Data masking is ideal for situations where reversibility is not essential, while traditional encryption provides the ability to restore ciphertext to its original plaintext. Format-preserving encryption (FPE) adapts to accommodate structured data stores, and tokenization replaces sensitive information with a randomly generated token. The choice of method depends on the specific needs of the business, including whether the data needs to be reversible, its format preserved, or encrypted. HashiCorp Vault's feature matrix can help identify the right method for each use case, allowing businesses to increase the usability and performance of their data protection workflow.

HashiCorp Consul and Nomad can utilize Azure managed identities to authenticate against Azure, enabling the elimination of hard-coded service principal information. The use of cloud auto-join allows for dynamic cluster joining without requiring static IP addresses or DNS names. This simplifies network automation for services and reduces operational complexity. By leveraging managed identities, Consul and Nomad can provide highly available service mesh capabilities that facilitate communication between services across regions. Additionally, the use of Azure managed identities with these tools offers a more secure approach to managing service principal information, as it eliminates the need to hard-code sensitive details.

The 2021 HashiCorp State of Cloud Strategy Survey reveals insights into how companies are spending on cloud and multi-cloud initiatives. While cost was a primary concern for many organizations, more than a quarter cited potential cost savings as a driver of cloud adoption. The survey found that cloud budgets vary significantly by industry, location, and company size, with telcos and financial firms being among the biggest spenders. However, complexity in tracking and controlling cloud spending is a significant issue, with 39% of respondents overspending their budgets in 2020. Despite this, many organizations can take steps to minimize cloud waste by automating, standardizing, and governing infrastructure provisioning, as well as investing in standardized tooling. The survey also highlights the importance of adopting a multi-cloud strategy and toolset for effective cloud management.

Terraform Cloud and Terraform Enterprise now offer new options to provide more explicit control over when resource state is refreshed and when resources need to be replaced, allowing customers to reduce the time it takes to make changes to their infrastructure. The new -refresh=false plan option enables teams to execute plans without a preemptive state refresh, while the -refresh-only plan mode allows for updating the state at a more convenient schedule. Additionally, the -replace planning option provides an improved way to replace degraded resources as a single atomic operation within a Terraform plan, avoiding potential race conditions in collaborative environments. These features are now available in both Terraform Cloud and Terraform Enterprise, enabling customers to streamline their infrastructure management workflows.

HashiCorp Terraform Cloud has introduced workspace tags to help organizations better manage their workspaces within the platform. These tags allow users to organize, filter, and correlate workspaces based on assigned tags, providing additional contextual awareness at the workspace level. The new feature is available in both Terraform Cloud and Terraform Enterprise, enabling users to easily identify and isolate specific workspaces using a "Filter by Tag" button. Administrators can also delete tags at the organization level and manage them through the Settings panel or API. This addition enhances the platform's ability to help organizations adopt infrastructure as code at scale, providing a more efficient way to manage workspaces.

The HashiCorp State of Cloud Strategy Survey reveals that 76% of enterprises currently employ a multi-cloud architecture, with over 9 in 10 planning to adopt this approach within two years. The survey also found that digital transformation is the top reason for choosing multi-cloud, cited by 34% of respondents, followed by portability (25%), scaling (25%), and delivering value to the business (19%). However, costs, security, and lack of in-house skills are major inhibitors to cloud adoption, with lack of skills being a top-three concern. The COVID-19 pandemic has accelerated cloud efforts, particularly for larger organizations, but its impact is nuanced, with only 7% citing it as a driver of multi-cloud adoption. Despite the challenges, over half of respondents believe that multi-cloud is helping them achieve their business goals, and large organizations are more likely to see benefits from this approach.

HashiCorp released Vault 1.8 with enhanced features and enhancements, including control groups that enforce additional authorization factors and troubleshoot server issues. The new version also introduced improved lease management, allowing Vault servers to stop attempting to revoke expired leases after a certain number of failed revocation attempts. Additionally, Vault Enterprise now autoloads the license as the server starts up, and there are tutorials available on HashiCorp Learn to help users explore these changes and learn more about control groups, troubleshooting irrevocable leases, and inspecting data in BoltDB.

HashiCorp has released Waypoint 0.5, which focuses on improving visibility into deployment health and enabling teams to perform more actions in the Waypoint UI. The new version introduces governance features such as status reports, input variables, OIDC authentication and user accounts, and a new CLI command called `waypoint status`. These features aim to provide better traceability, customization, and authentication for developers deploying applications across platforms. With Waypoint 0.5, teams can now view deployment resources and their health status in real-time, track the health of physical resources created as part of its operations, and access a breakdown of deployment resources such as Kubernetes Services and their current health status from the CLI and UI.

The inaugural cloud strategy survey reveals that multi-cloud adoption is now the new normal, driven by digital transformation and influenced by skill shortages, security concerns, and COVID-19's impact on the economy. 76% of respondents are already using multi-cloud environments, with larger enterprises being more likely to adopt this approach. Digital transformation initiatives are a top driver of multi-cloud adoption, with 34% citing it as one of their top-three business and technology drivers. However, skill shortages remain a significant challenge, with over half of respondents citing it as a top-three inhibitor. Security is also a key concern, with 47% citing it as a top-three cloud inhibitor, but security issues can also drive cloud adoption forward in some organizations. The pandemic has had a complex impact on cloud adoption, with many organizations already shifting to multi-cloud before the pandemic and others seeing it accelerate their digital transformation initiatives.

Kubernetes and HashiCorp Vault are two mature orchestrators used for managing the lifecycle of containerized applications. Kubernetes' native secrets management functionality is limited, as it stores sensitive information in etcd without encryption or a time-to-live, making it vulnerable to security breaches. In contrast, HashiCorp Vault provides a centralized secrets repository that encrypts data by default during transit and at rest, with built-in mechanisms for secret renewal, rotation, and revocation. Vault integrates with Kubernetes through authentication and secret retrieval, allowing users to delegate the responsibility of creating and managing the lifecycle of a secret to Vault. Nomad, another orchestrator, focuses solely on cluster management and scheduling, delegating secrets management to Vault or external providers. The integration between Nomad and Vault enables simplified operations for users, while Nomad's flexibility allows for managing jobs and their secrets across a heterogeneous environment.

HashiCorp has released Boundary 0.5 and Boundary Desktop 1.2.1, which includes a new event logging system that provides operators with fine-grained visibility into running Boundary clusters. The new system allows for filterable event logs by type and user-defined expressions. Additionally, the release introduces CRUD capabilities for credential stores and libraries in the administrative console, as well as enhancements to session cleanup. These changes are available for download now, and users are encouraged to review the upgrade guide and Release Notes before upgrading.

HCP Consul is designed to enable developers and operators to focus less on managing underlying infrastructure and concentrate more on enabling Consul's service discovery and service mesh capabilities. By deploying production-ready Consul servers with the click of a button, users can get greater insights into how applications registered with HCP Consul clients perform using application performance monitoring solution Datadog. The integration allows users to configure Datadog to trigger alerts based on changes to latency data, service health checks, and node health checks, providing full visibility into service changes and health without worrying about parsing through server-side data. Additionally, the integration with Envoy enables users to ingest request volume, traffic flow, and span data to build a comprehensive visualization of how traffic moves through the Consul service mesh, making it easier to troubleshoot errors and reduce mean time to resolution. By leveraging HCP Consul and Datadog, organizations can improve observability dashboards and application performance without spending time on routine maintenance and management of Consul servers, allowing them to focus on enhancing their businesses.

I've tried out the latest tutorial on how to create a custom Chrome extension to fetch secrets from Vault, and it was an interesting exercise. As the first engineer on the Vault Developer Experience team, I was eager to gain hands-on experience with this tutorial and understand the pain points that developers might face when working with Vault. The tutorial provided a clear and concise guide on how to create a basic Chrome extension that can fetch secrets from Vault using the `vault` command-line tool. By following along with this tutorial, I gained a deeper understanding of the potential use cases for custom Chrome extensions in conjunction with Vault, such as securely storing sensitive data or providing a secure way to access configuration variables. Overall, I found the tutorial to be well-structured and easy to follow, making it accessible to developers who are new to both Chrome extension development and Vault.

HashiCorp's Vault is a secrets management platform that stores static secrets and provisions dynamic, short-lived secrets. To reduce operational overhead associated with Vault cluster availability, it can use Microsoft Azure managed identities to automatically obtain the unseal key from Azure Key Vault. This eliminates the need for operators to manually unseal Vault after server restarts or patching operations. The process allows Vault to authenticate against Azure and gain access to the unseal key without requiring sensitive parameters like client ID and secret in the seal stanza, making it a more secure and streamlined solution.

The Kubernetes manifest resource has graduated from beta to official support, allowing users to manage custom resource definitions (CRDs), custom resources, and other unsupported Kubernetes resources using Terraform. This new resource provides a pure Terraform way of declaring arbitrary Kubernetes manifests in HCL, enabling dynamic management of Kubernetes API resources without relying on additional providers. The resource can be used to create any Kubernetes API resource, including CRDs, and is recommended for converting YAML manifests to HCL. Users who were already using the kubernetes-alpha provider alongside the official Kubernetes provider can simply remove it and use the new manifest resource instead. The resource has limitations, such as requiring access to the Kubernetes API at plan time and slower performance compared to named resources in the provider. Future work includes adding import support, greater flexibility in waiting for conditions and events, and a data source analog to the manifest resource.

The HashiCorp Cloud Platform (HCP) has introduced the `Consul Plus` offering, which allows users to federate Consul clusters across multiple regions for improved redundancy and resiliency of applications. Federation simplifies communication between different datacenters, reducing operational burden and lowering risk by reducing ingress/egress points through a single gateway. The new offering enables multi-region deployments and supports recovery and resiliency, end user support across regions, and overlapping IP ranges, making it easier for organizations to build more resilient applications and address locality-aware deployment needs.

HCP Vault Starter is a new fully managed HashiCorp Vault offering now available on AWS, providing a production-grade 3-node cluster at a reduced price point for customers with modest scale. It offers a similar feature set to the Standard cluster but with a limit of 25 clients and is priced lower, making it an attractive option for organizations looking to secure sensitive data and manage secrets in their AWS environments. The Starter cluster can be easily deployed using HashiCorp's push-button deployment and fully managed infrastructure, providing resilience and operational excellence as a service. With its pricing options ranging from $0.03 per hour for the Development node to $7.489 per hour for the Standard cluster, HCP Vault provides a flexible solution for organizations of all sizes, allowing them to get started quickly and easily with secrets management and encryption capabilities.