November 2020 Summaries

HashiCorp Consul 1.9 is now generally available, providing more granular control for service mesh users, improving observability into services within the service mesh, and delivering tighter integrations with runtime platforms in a way that is more Kubernetes-native. The release includes features such as Application-Aware Intentions, Service Mesh Visualization, Custom Resources for Kubernetes, Deploy Consul in OpenShift, Active Health Checks for Consul on Kubernetes, and Streaming. These new features aim to provide increased control over and observability to services within the service mesh, enabling users to adopt a service mesh in more environments and via more developer-friendly ways. The release is available now, and users can download the binaries or refer to the Consul 1.9 changelog for a detailed list of changes.

The Terraform AWS provider has introduced support for Code Signing, a trust and integrity control feature that allows users to verify the source and integrity of AWS Lambda functions. This feature leverages AWS Signer, a new fully managed code signing service, enabling administrators to create digitally signed code artifacts and verify their signatures at deployment. With Code Signing, developers can ensure that only unaltered code is published by approved developers within their Lambda functions, reducing the risk of tampering or alteration during deployment. The feature also allows for separation of responsibilities between operators and developers, promoting a more secure and agile development process.

HashiCorp will be hosting a Hackstravaganza event, `HashiCraft Holidays Hackstravaganza`, where participants can showcase their creativity by combining HashiCorp products with arts & crafts to create holiday-themed projects. The event is open to teams of up to 4 people and requires registration on the Discuss platform. Participants must submit a video showcasing their project, explaining its purpose, appearance, execution, and communication. The contest will run from November 20th to January 4th, with winners announced in the first two weeks of the new year based on creativity, aesthetics, functionality, and storytelling.

The Nomad 1.0 release includes a new feature called Event Stream, which provides stronger tracing and debuggability for Nomad clusters. This feature allows operators to view and subscribe to a single unified timeline that streams all high-level events to better understand how the cluster is performing. The Event Stream replaces blocking queries, which had drawbacks such as no guarantee of a change or requiring coordination on the client to monitor multiple types of requests. With the Event Stream, operators can easily subscribe to changes within a cluster through a single endpoint and request. The feature creates a set of events when a state change occurs in Nomad, which are then pushed onto the server's event broker. The event broker must be enabled by setting enable_event_broker in the server's configuration block. Operators can access the Event Stream using an HTTP client or programmatically using the Nomad API package. The feature allows for filtering on certain topics and provides a way to subscribe to all events or specific events. The Go package to use Nomad's API makes it easy to subscribe to the event stream from a Go application.

Distributed tracing, a key component of observability, helps understand complex systems by representing the end-to-end request flow through distributed systems. Observability tools come in two varieties, DIY (open-source build-it-yourself) and commercial offerings, each with its pros and cons. While open-source tools provide a simple way to start monitoring applications and services, they require dedicated engineers for maintenance and updates. Commercial products alleviate this burden, providing out-of-the-box dashboards and alleviating the need for engineering teams to maintain their own observability stack. In addition, as application delivery models evolve from monolithic apps to microservices, distributed tracing solves the challenge of observability in these complex systems by representing individual units of work done in a distributed system, known as spans. Consul Service Mesh provides service-to-service connection authorization and encryption using mutual TLS, helping secure services and provide data about service-to-service communications. By leveraging the observability features of Envoy proxy or instrumenting code directly, developers can apply distributed tracing to achieve one of the Three Pillars of Observability.

The Terraform AWS provider has added support for the newly released AWS Network Firewall service, which provides essential network protections for Amazon Virtual Private Clouds (VPCs) with just a few clicks and scales automatically. The service allows fine-grained control over network traffic through its flexible rules engine, enabling users to define firewall rules that block malicious activity and integrate with managed intelligence feeds sourced by AWS partners. To use this feature in the Terraform AWS provider, users need to employ three new resources: `aws_networkfirewall_firewall_policy`, `aws_networkfirewall_firewall`, and `aws_networkfirewall_rule_group`, along with additional attributes to existing resources. A sample Terraform configuration is provided to demonstrate how to configure an AWS Network Firewall VPC, firewall policy, and firewall rule group. The feature can be used in conjunction with AWS Firewall Manager to build policies based on AWS Network Firewall rules and centrally apply them across VPCs and accounts.

The blog discusses the growing number of customers choosing Nomad as an alternative to Kubernetes for orchestration, highlighting its two unique core strengths. Nomad excels in simplicity and flexibility, providing ease-of-use on-premises, at the edge, and in the cloud. Its lightweight binary design enables companies to scale and manage an orchestrator with minimal operational overhead. Organizations are capable of translating this simplicity into real business results, with customer interviews showing that teams can go from a technical proof of concept to production in 1-3 weeks. Nomad's flexibility also allows companies to bridge legacy applications with modern orchestration capabilities without rewrites, making it an attractive option for those looking to maintain existing workloads while adopting new technologies. Additionally, the blog touches on the ecosystem aspect, noting that Nomad is building a simpler and more prescriptive path to the ecosystem by integrating with leading partners and investing in significant improvements. Medium- to large-sized enterprises often adopt multiple orchestrators, leveraging each scheduler's strengths to supplement their needs. The blog concludes that the multi-orchestrator deployments will continue to emerge as a pattern, where teams use Nomad and Kubernetes in areas where their specific strengths are needed.

The introduction of new dependency lock files in Terraform 0.14 aims to simplify managing Terraform automation by ensuring that provider versions are consistently applied during automated workflows. The lock file is generated automatically when `terraform init` is run and should be committed into version control systems for future runs to guarantee the same provider versions. Upgrading to a new provider can be completed via `terraform init -upgrade`. The lockfile entry includes hashes of provider packages, which must match exactly between runs to replicate previous results accurately. If not using this new behavior, users can run `terraform init -upgrade` and discard `.terraform.lock` immediately after it is created, or add the lock file to a list of files ignored by version control. Terraform 0.14 also includes several other features such as Getting Started guides, tutorials, and improved security measures like Protecting Sensitive Input Variables.

Consul 1.9 now provides OpenShift support for Kubernetes deployments with a secure-by-default configuration, allowing users to easily deploy Consul Enterprise onto OpenShift to provide an enterprise-grade multi-cluster service networking experience. This integration supports RedHat OpenShift v4 and is made possible through the use of Helm3, which simplifies the deployment process and ensures security settings meet OpenShift's requirements. Users can install Consul on OpenShift by creating a new project, adding the HashiCorp Helm releases repository, and running the Helm chart with specific flags or values config file. Additionally, Consul for Kubernetes can be configured to allow external Consul servers to connect to a remote Kubernetes cluster, providing flexibility in managing Consul clusters outside of OpenShift.

The latest version of Consul, 1.9, introduces Custom Resource Definition (CRD) support, allowing users to interact with Consul in a more Kubernetes-native experience. This enables users to configure L7 routing, intentions, service configuration, and proxy defaults using Kubernetes native manifests, making it easier to manage a service mesh on Kubernetes. CRDs are modeled after Configuration Entries, which allow defining configuration at a cluster or namespace level, reducing the need to switch between Consul CLI and kubectl CLI tools for configuration. The release demonstrates how to set up L7 routing and intentions using CRDs to route traffic correctly between services, including splitting traffic between two versions of a service. Additionally, users can manage intentions using a ServiceIntention CRD, enforcing access control based on HTTP path prefix and methods made for a request from one service to another, with the release also introducing support for managing custom resource definitions using kubectl, allowing for CRUD operations.

HashiCorp is a sponsor at the KubeCon + CloudNativeCon North America Virtual conference, showcasing its product set and participating in various activities to engage with attendees and provide technical information on how HashiCorp tools integrate with Kubernetes and other CNCF projects. The company's products, including Terraform, Vault, Consul, and Nomad, are designed to work seamlessly with these projects, enabling users to manage infrastructure, security, networking, and application deployment more efficiently. HashiCorp has made significant progress in integrating its products with Kubernetes, including the release of new providers and tools for Terraform and Helm, as well as improvements to Consul's service discovery and observability capabilities. The company is committed to strengthening the value delivered by these products to the Kubernetes community, particularly through recent product announcements that enhance their interaction with Kubernetes as a platform.

The release of Consul 1.9 is a significant update aimed at enhancing the experience of running Consul on Kubernetes, providing features such as support for Custom Resource Definitions (CRDs) to manage resources within environments, improved OpenShift support, new service mesh visualization capabilities in the UI, and support for Kubernetes Readiness probes to enable real-time health data and traffic management. These updates address key gaps identified by users and create a more Kube-native experience for developers, while also providing value to users managing a service mesh across multiple platforms.

HashiCorp has released Nomad 1.0, which includes Dynamic Application Sizing (DAS), a new feature in Nomad Enterprise that optimizes application resource consumption at scale without manual trial-and-error. DAS aims to reduce toil by automatically monitoring jobs and providing recommendations for resource limits, maximizing infrastructure usage, and improving reliability. The feature builds on top of Nomad's existing Autoscaler capabilities and provides a Recommendations API, vertical autoscaling policies, and DAS-specific plugins. With the new feature, users can start a free 30-day trial or reach out to sales for more information.

HashiCorp released Vault 1.6 on November 11th, introducing new features and enhancements for cloud provider key lifecycle management, GitHub Actions, data transformation, usage metrics, integrated storage, and password policies for databases. The Key Management Secrets Engine is a Technical Preview feature designed to help with cloud provider key lifecycle management, while the Tokenize Data with Transform Secrets Engine tutorial demonstrates a new method introduced in Vault 1.6 for replacing sensitive data with unique tokens. Vaults UI now supports creating transformations, roles, templates, and alphabets for the Transform secrets engine, and the Vault Usage Metrics tutorial introduces the Vault Metrics dashboard to help discover active client count and historical client counts.

HashiCorp Terraform Cloud has been selected as a finalist in the 2020 CRN Tech Innovator Awards for Cloud Tools & Management, a prestigious award recognizing innovative vendors in the IT channel. The platform provides infrastructure automation as a service and has gained significant traction with over 5,000 new users per month since its initial release in 2019, executing over 500K runs each month and managing over two million resources. Terraform Cloud offers numerous benefits to organizations, including accelerated cloud adoption, improved security, compliance, and operational consistency, as well as increased flexibility and reduced costs through standardization on a single platform. This award is not only a recognition of HashiCorp's innovative solution but also an opportunity for partners to expand their offerings with Terraform Cloud, making it an attractive door-opener for partner sellers and a bridge to other HashiCorp products like Vault and Consul.

This release of HashiCorp Vault 1.6 introduces several key features and improvements, including Integrated Storage enhancements, a new Key Management Secrets Engine, Transform Secrets Engine updates, and more. Integrated Storage now offers Cloud Auto-Join, which allows nodes to automatically join a Vault cluster in cloud environments. The Transform Secrets Engine has been updated to include tokenization support, allowing for the replacement of sensitive data with unique non-reversible values. A new Key Management Secrets Engine is also introduced, enabling the management and secure distribution of keys to various cloud KMS services. Additionally, this release includes many other new features, workflow enhancements, general improvements, and bug fixes, making it a significant upgrade for users.

Terraform 0.14 introduces the ability to define values as "sensitive", making it easier to redact sensitive information in Terraform workflows, allowing users to thread this notion throughout Terraform, and enabling provider-schema-based sensitive values, which propagates the sensitive mark through the plan, potentially introducing redacted values where expected by the end user.

The HashiCorp team has developed a zero-configuration topology visualization tool for Nomad 1.0, which helps operators stay on top of their clusters by providing an overview of the cluster's state and allocation distribution. The tool is designed to optimize density, avoid duplicating existing features, and encourage exploration of the cluster state. It uses data from Nomad already has, is zero-configuration, and starts somewhere. The visualization allows users to see all allocations organized, understand job placement decisions, and associate allocations with jobs and task groups.

Sentinel 0.16 introduces several feature enhancements that improve the development experience for policy authors. The release unifies the policy authoring workflow by including HCL support and expanding the standard imports, allowing for standardized policy configuration syntax, central policy management, and new import in the Sentinel standard library. This update enables easier evaluation of policies, simplifies the process of managing policies from a central source, and supports definitions of remote sourced policies and modules, which encourages code reuse. Additionally, a new version import is included, providing policy authors with the ability to parse versions and version constraints, supporting both arbitrary versions and versions that follow the semver.org specification. The update aligns Sentinel with other HashiCorp products and provides seamless authoring, mocking, and testing of Sentinel policies.

HashiCorp Boundary is an open source project that enables users to securely access dynamic hosts and services with fine-grained authorization, designed to manage session connectivity to infrastructure in a controlled manner. It uses scopes as containers to isolate permissions boundaries, creating defined blast radii for mitigation of identified risks during threat modeling exercises. Scopes can be thought of as levels within the domain model, starting from global scope at the top level to project scope at the lowest level, with organizations and projects serving as child scopes in between. Boundary approaches Identity and Access Management (IAM) by creating principals such as users and groups, assigning capabilities through grants, roles, and authentication methods. The system's access management starts at the bottom level with Actions, which can be specified in Grants, and Roles are a collection of zero or more grants that belong to a single scope. Boundary integrates with Terraform provider for automation and management of infrastructure state.

The HashiCorp team has released two new libraries for Terraform provider development, enabling more powerful interactions with Terraform. The `terraform-plugin-go` module provides a lower-level abstraction of the Terraform protocol, allowing advanced developers to access all of Terraform's functionality at the expense of fewer guardrails and more verbosity. The `terraform-plugin-mux` module allows provider developers to multiplex multiple provider implementations into a single Terraform provider server, enabling easier upgrades between frameworks one resource at a time. Additionally, the Terraform Plugin SDK has been migrated to use `terraform-plugin-go`, making it transparently usable with `terraform-plugin-mux`. These new modules aim to improve the Terraform provider developer experience and enable more aggressive development of enhancements for new resources without breaking existing ones.

Terraform 0.14 introduces an experimental, on-by-default concise diff renderer to improve the understanding of changes made by Terraform plans. The new renderer hides unchanged and irrelevant fields, only showing changed or added values in unordered collections and structural types, and providing context around changes in sequences. This change aims to simplify diffs for practitioners and make it easier to reason about actual changes, especially when dealing with large diffs resulting from slight changes to existing resources. The diff renderer also includes features like displaying hidden attribute counts and nested block totals at the end of each resource. With this update, Terraform 0.14 offers improved support for collaborative workflows and solidifies its ecosystem, empowering organizations of all sizes to adopt the platform effectively.