July 2020 Summaries

The Terraform AWS Provider has undergone significant changes with the release of version 3.0, introducing four major enhancements: updated Amazon Certificate Manager resources, removal of hashing from state storage, improved authentication ordering, and deprecation of Terraform 0.11. These changes aim to simplify configurations and improve the overall experience for users, while also consolidating example code to use Terraform 0.12 syntax. The provider team has worked hard on these improvements and invites feedback through GitHub Issues, with a recommended pinning of provider versions to protect against unexpected circumstances.

HashiCorp's Consul allows operators to quickly connect applications across multiple clouds and runtime environments, enabling a global service mesh approach that supports workflows and enables security policies. Consul provides features such as encryption with mTLS, zero trust networking, and federation of workloads between cloud environments. The platform is designed to make multi-cloud networking easy for operators, allowing them to deploy a standalone cluster or integrate with existing clusters. Consul's deployment on Kubernetes using Helm charts enables flexible configuration options, including automatic bootstrapping and configuration of Access Control Lists, federation secrets, and mesh gateways. By deploying workloads across cloud environments, organizations can craft failover domains and progressive delivery patterns, while leveraging traffic shaping policies to control the flow of traffic. Consul's service mesh design pattern enables a network-first approach, where applications are treated as first-class citizens on the network, and connectivity is applied dynamically based on application requirements.

The concept of cloud bursting, which was initially touted as a way to expand on-premise data center capacity with the elasticity of the cloud, has proven to be complex and challenging to implement due to networking complexities and security implications. However, in today's scenario where internet usage is increasing significantly, particularly in streaming platforms and video conferencing tools, companies still need to scale up their capacity to handle sudden spikes in load. Cloud bursting can provide a solution for these organizations that are not yet geared towards cloud computing, offering a more efficient use of resources than building out hardware or migrating to the cloud immediately. The proposed setup uses service mesh products like Consul and Nomad to securely connect on-premise and cloud environments at scale, allowing for dynamic routing of traffic between applications in different data centers.

The Principle of Least Privilege is a key concept in information security, emphasizing that team members should only have access to privileges required for their tasks. As organizations using Terraform grow, implementing this principle becomes increasingly important and can be challenging. To address this, Terraform Cloud has introduced Custom Workspace Permissions, allowing organization owners to specify granular permissions for each workspace using a new UI. This feature enables mixing and matching of existing permissions, restricting access to State Versions, and setting customized permissions for teams. It's essential to note that while this feature enhances security, it should be part of a larger defense-in-depth approach and complemented with strong oversight and CI/CD processes.

Terraform 0.13 introduces powerful new meta-arguments for modular workflows, including count, for_each, and depends_on, which can simplify and streamline Terraform configurations at the resource-level. These features allow modules to produce multiple instances systematically, enabling a concise way to declare shared dependencies between module objects in the calling module. The new features are designed to make configuration easier to maintain and allow Terraform to maximize concurrency when making many changes in a single operation. The `depends_on` meta-argument is available for module-centric workflows and can be used to imply dependencies using data flow, but its use is recommended as a last resort.

HashiCorp has announced the release of its official Linux repository, providing Debian and RPM packages for HashiCorp products such as Vault, Consul, Nomad, Terraform, Packer, and Vagrant. The repository aims to improve the installation and upgrade experience for Linux users by offering a centralized source for these products. Currently, supported versions are available, with new releases published on releases.hashicorp.com. Users can easily add the repository to their system using various package managers, including apt, yum, and dnf. The packages include binaries, configurations, unit files, and install/uninstall hooks, designed to facilitate a straightforward installation and upgrade process. Future improvements will expand distribution and architecture support, with continuous community feedback aimed at providing the best possible experience for users.

HashiCorp Consul Service (HCS) on Azure is now generally available, enabling teams to provision HashiCorp-managed Consul clusters directly through the Microsoft Azure portal. This allows for easy access to various Consul use cases, including service discovery, automated network configuration, and secure service-to-service communication with service mesh. HCS supports both development and production clusters, as well as on-demand development clusters that can be used to evaluate key aspects of the service or execute a proof of concept. The service also includes an Azure CLI extension for streamlined integration and management. Additionally, HCS on Azure supports HashiCorp Consul 1.8 clusters by default, with new features like ingress gateways, terminating gateways, and single sign-on that enable extended and integrated deployment of enterprise service meshes. Furthermore, the service will soon support WAN federation over mesh gateways, allowing for simplified network configurations across multiple environments. Several key partners have tested and support their existing Consul integrations with the GA version of HCS on Azure.

Vault 1.5 has been announced, bringing new features such as improved security, performance tuning guidelines, and enhanced monitoring capabilities. The Vault Learn site offers tutorials on various topics including installation on RedHat OpenShift, performance tuning, monitoring telemetry data with Splunk, integrated storage for high availability coordination, protecting vaults with resource quotas, user configurable password generation, codifying management of vaults, SSH secrets engine one-time passwords, and Vault Enterprise replication UI enhancements. These updates aim to improve the overall stability and usability of the Vault platform, providing users with better tools to manage their secret storage needs.

HashiCorp has released HashiCorp Vault 1.5, a tool that provides secrets management, data encryption, and identity management for any application on any infrastructure. The new release focuses on improving Vault's core workflows and integrations to better serve user needs. Key features include the addition of telemetry metrics and a Splunk monitoring app, expanded support for running Vault in Kubernetes via OpenShift, and resource quotas to protect against distributed denial of service (DDoS) attacks. Additionally, Vault 1.5 introduces integrated storage as HA storage, improved replication UI, and certifications with VMware and NetApp. The release also includes various new features, workflow enhancements, and bug fixes. Users are encouraged to upgrade and test the release in an isolated environment before deploying it to production.

The HashiCorp and Splunk collaboration has led to the launch of a new Splunk app that helps enterprises monitor and analyze HashiCorp Vault's operational and security performance in a multi-tenant environment. The app provides pre-built dashboards and reports, including metrics on CPU usage, garbage collection, token creation, entity identity, secret storage, and more. It also includes an updated guide for monitoring Vault telemetry and audit device log data with Splunk, which offers recommendations on key metrics to monitor, threshold values, and how to estimate normal ranges. The app is available with Vault Enterprise, but all data sources are accessible with all versions of Vault, allowing users to construct the dashboards from scratch using step-by-step instructions provided in the guide.

HashiCorp Vault 1.5 introduces Resource Quotas to protect against distributed denial of service (DDoS) attacks and ensure stability and resource consumption are predictable, allowing operators to control how applications request resources from Vault using rate limit quotas and lease count quotas. These features enable canceling requests over a set rate to maintain the overall health of Vault, protecting it from misbehaving applications that might saturate resources in the cluster through high request rates. Operations staff can monitor Resource Quotas using audit logging and enhanced telemetry metrics, such as quota.rate_limit.violation, quota.lease_count.violation, and quota.lease_count.max, to detect rate limit quota rule violations and ensure a healthy cluster for everyone else.

HashiCorp has updated its Vault Helm chart to support RedHat OpenShift 4.X, enabling users to install and run Vault Enterprise on OpenShift. The recommended installation method involves adding the Hashicorp helm repository, installing the latest release of the Vault Helm chart, and configuring parameters as needed. The Helm chart supports various deployment modes, including Dev mode, Highly Available Raft Mode, and External mode. To get started, users can access Learn Guides, documentation, and video demos that showcase the integration of Vault with OpenShift. HashiCorp continues to add new features and improvements to its product, and is hiring for positions related to its technology.

Vault enterprise feature Replication is a key component with two main use cases: Disaster Recovery (DR) and Performance. Its core unit is a Vault cluster, consisting of a primary cluster linked to follower secondaries that communicate in near real-time. The new replication UI features redesigned dashboards with improved monitoring capabilities, making it easier to scan for potential issues and troubleshoot problems. These changes include cards displaying state, last_wal, and known_secondaries, as well as new information such as secondary ids and connection states. A custom "summary" dashboard is also available for clusters that serve both DR and Performance primary roles, providing an overview of both and linking to individual dashboards. The development process involved collaboration across Product, Design, and Engineering teams, with input from various stakeholders and customers, resulting in a new iteration of the replication UI.

The HashiCorp Cloud Engineer Certification program aims to provide a baseline of knowledge for individuals using Terraform, Vault, and other HashiCorp products. Technical certifications can prove knowledge about a technology and serve as a structured pathway to learning a new technology. The certification program has released the Terraform Associate and Vault Associate exams, which are one hour long and predominantly multiple-choice tests taken online with a proctor. The Consul Associate exam is currently in beta. To prepare for the exams, it's recommended to familiarize yourself with the exam objectives, use training materials such as the official review guides and practice exams on platforms like Udemy, and try out the technology in a production-like environment.

HashiCorp has released an update to its Azure provider for Terraform, which simplifies deployment and management by defining components as code. The new version 2.0 includes Sentinel policies as code to enforce security, regulatory compliance, and internal business policies. A preview of the Terraform Foundational Policies Library is also available, offering 50 initial policies based on Center for Internet Security Benchmarks for major cloud providers. These policies can be applied to Azure configurations using a video tutorial, combining with existing Sentinel rules and checked during every Terraform plan operation.

The Cloud Development Kit (CDK) for HashiCorp Terraform allows users to define infrastructure resources using TypeScript, Python, and other supported programming languages. This enables the use of familiar programming languages with Terraform's extensive provider ecosystem. The CDK generates Terraform configuration files in JSON format that can be applied with `terraform apply` or used with the CDK for Terraform CLI. Users can also synthesize Terraform configuration from TypeScript code using `cdktf synth`. The project is in alpha and supports additional languages, such as JavaScript, Java, and C#. It aims to provide a user experience similar to the AWS CDK, with first-class commands and expanding support for other providers and modules within the Terraform Registry. The community is encouraged to provide feedback through GitHub issues, pull requests, and the HashiCorp Discuss form.

** The new provider source attribute in Terraform 0.13 allows users to declare the registry source of a Terraform provider, creating a clear workflow for using providers from the Terraform Registry. This attribute is part of the required_providers setting inside a terraform configuration block and enables automatic download and installation of partner and community providers. The provider source string is made up of the hostname, namespace, and type, with optional hostname and namespace values that can be omitted or set to default values. Local names can be declared for multiple providers with the same type to easily identify them in the configuration. Third-party providers need to be assigned an arbitrary source and placed in a specific directory hierarchy for Terraform to find and use them. Users will need to upgrade their configurations by adding required_providers entries for non-HashiCorp-owned providers. The new feature aims to benefit Terraform users and provider developers alike, making it easier to include community and partner providers in the core terraform init workflow.

]]Consul Enterprise offers several features to help organizations build resilient platforms for business-critical applications in the cloud. These include proactive measures such as automated upgrades that minimize downtime and enhance read scalability by deploying additional capacity, advanced federation that enables independent datacenter operations with reduced latency, network segments that restrict LAN gossip within a datacenter, and redundancy zones that enable failover capabilities to ensure minimal data loss in case of outages. Additionally, Consul Enterprise provides automated backups to capture current data and restore it in the event of data loss or corruption. By implementing these features, organizations can position themselves to recover quickly and efficiently from unexpected events.

Terraform is a powerful tool for creating and managing greenfield infrastructure, as well as managing existing resources from popular infrastructure providers like cloud-based databases. It allows users to migrate manually created infrastructure into its workflow, enabling them to manage the entire lifecycle of their project with Terraform's configuration language and workflow. Users can import existing resources into a Terraform project's state using the terraform import command, allowing them to start managing those resources without destroying or recreating them. This process requires creating additional configuration to match the state created by the import process.

Nomad 0.12 introduces significant networking improvements, including multi-interface networking and CNI plugin support, which allows cluster administrators to define CNI plugins available on a node for job authors to use, and enables the selection of which network a job requires. This feature is useful in cases such as ingress services that route traffic from the public Internet to internal services, where multiple logical networking interfaces can be utilized. The improvements also address security concerns with host networking and task driver-based workarounds, allowing Nomad to enforce more secure networking configurations. Existing clusters and workloads will continue to function as usual, and cluster administrators can leverage these new features to enhance network configuration options for job authors.

Thomas Weber, CTO of pascom, a company that develops next-generation UCC Telephony Solutions and upgrades business communications, has been searching for ways to improve its solutions without disrupting existing workflows. He identified two ideal platforms, HashiCorp Nomad and Podman, which would provide a technological solution to achieve his goals. However, the challenge was that these platforms do not talk to each other out of the box. To address this issue, Weber developed an open-source research project to create a prototype plugin that enables the combination of Nomad and Podman within a cloud infrastructure. The plugin is now available for download and experimentation, and Weber hopes it will provide the basis for future implementations and deployments, as well as support the wider community who are looking to take advantage of the excellent benefits offered by both technologies.

HashiCorp has released Nomad 0.12, a significant update to its orchestrator tool that offers improved deployment flexibility and operator maintainability at scale. The new version introduces Multi-Cluster Deployment, which enables users to deploy applications seamlessly across federated clusters with configurable rollout and rollback strategies. Spread Scheduling allows for even load distribution across the cluster, while Container Network Interface (CNI) supports deploying containerized applications with desired network configurations from third-party vendors. Other notable features include Soft Memory Limits, Windows IIS Task Driver, Podman Task Driver, Cluster Autoscaling, Nomad Snapshot, Nomad Debug, Cross-Namespace Queries, Automated Backups, and Global Search, Job Scaling, and Nomad Monitor in the UI. These updates enhance Nomad's capabilities for production use cases, making it a more attractive option for organizations looking to manage their containerized applications at scale.

HashiCorp has released a new feature in Nomad Enterprise 0.12 called Multi-Cluster Deployment, which enables the deployment of applications across multiple clouds and datacenters with a single unified monitoring and control plane. This feature allows organizations to deploy applications seamlessly to federated Nomad clusters with configurable rollout and rollback strategies, achieving high availability, built-in flexibility, no setup needed, and fully supported by HashiCorp. The new feature simplifies the process of deploying multi-cluster applications without requiring additional namespaces, topology, or configuration changes. It also provides a more coordinated approach to rolling out deployments across regions, allowing for automatic blocking and unblocking of regions as needed. Additionally, the feature includes support for failure handling, including the option to allow a region to fail all remaining regions or only mark itself as failed.

The Nomad Autoscaler has been released in beta, offering horizontal application autoscaling capabilities and now also includes cluster autoscaling with support for AWS Autoscaling Groups. The new feature allows users to automatically add or remove clients from their Nomad cluster as their load changes, with a long-awaited capability that was previously only available through manual task group count management. Cluster scaling policies can be specified in files using the HCL syntax and include multiple check blocks for more complex metric values, while also supporting external plugins such as Prometheus and native Nomad metrics. The autoscaler will run these queries and pick the most appropriate result to make scaling decisions, making it easier to manage infrastructure with more complex requirements.

The 2020 HashiCorp community conference, HashiConf Digital, was held online due to the COVID-19 pandemic, attracting over 6,600 attendees from 105 countries who gathered for three days of product announcements, workshops, and education on HashiCorp products. The digital event featured high-quality technical sessions led by HashiCorp Co-Founders and CTOs, as well as new flagship cloud offering - the HashiCorp Cloud Platform. Notable highlights included emcees Domi and Rob, a fireside chat with Kelsey Hightower, digital meet-ups, and music intermissions. The event also featured a custom-built platform with Q&A cards, live chat, and on-demand content, setting the stage for future events like the October 12-15, 2020 edition of HashiConf Digital.

Terraform 0.13 has introduced custom variable validation, a production-ready feature that allows configurations to contain validation conditions for specific variables. This feature builds upon the type system in Terraform by providing a way to define error messages and conditions for variable values. The validation is evaluated in a reduced context and can be distinguished from errors caused by other parts of the configuration. With this feature, users can ensure data consistency and accuracy in their Terraform configurations.