April 2024 Summaries

OpenTofu 1.7 has been released, offering new features and improvements for infrastructure as code management. Highlights include state encryption, import blocks with for_each, provider-defined functions, a removed block feature, and new functions such as templatestring, cidrcontains, base64gunzip, urldecode, and issensitive. The S3 backend now aligns more closely with AWS CLI and SDKs in terms of credential search behavior. OpenTofu Commercial Support is available through Spacelift for those seeking assistance with migration or support coverage.

The article discusses the use of the `aws s3 cp` command, which is used to transfer data between local filesystems and Amazon S3 buckets efficiently. It covers various aspects such as common use cases for copying files to and from an S3 bucket, using option flags with the base `aws s3 cp` command, and how it differs from `aws s3 sync`. The article also provides examples of how to use the command in different scenarios, including uploading a local file stream to S3, downloading an S3 object as a local file stream, and copying files between two S3 buckets. Additionally, it explains how to set fine-grained grants on the files being copied using the `--grants` flag and specifying the storage class for the files being copied with the `--storage-class` flag.

Listing Docker containers is essential for monitoring their health, retrieving IDs or names, and checking port bindings. This tutorial covers various techniques for listing containers with different options to improve efficiency in managing containers. The main use cases include checking running containers, retrieving container details, viewing port bindings, and monitoring health check results. Docker provides several commands like docker ps, docker ps -a, docker ps -l, docker ps -n, docker ps -q, and docker ps --no-trunc for listing containers with different options. Additionally, the --format flag allows customization of container list output, while the --filter flag enables filtering by key/value field pairs. Docker also offers modern aliases like docker container list and docker container ls for listing containers and Compose projects, respectively.

CreateContainerConfigError and CreateContainerError are two common error messages that occur when deploying Pod or similar objects in Kubernetes due to invalid configuration properties or unmet conditions. Debugging these errors can be confusing, but understanding their causes and learning how to fix them can help keep your cluster healthy. Common causes include missing ConfigMaps and Secrets, naming conflicts, missing command/entrypoint specifications, inaccessible storage volumes, and container runtime issues. To resolve these errors, you can check the Pod's events, find the "Failed" event that reveals the cause of the error, troubleshoot the issue by editing your manifest, and reapply your Pod to your cluster once the problem is fixed. By following these steps, you can detect and solve these errors to ensure a smooth Kubernetes deployment process.

The article discusses the 'contains' function in Terraform, which checks if a specified value exists within a given list or set. It returns true if the value is found and false otherwise. The syntax of the function is contains(list, value). Practical examples using lists and strings are provided. The article also compares it to the strcontains function, which checks whether a substring is within another string. Both functions are useful for decision-making based on list or set contents and string manipulations respectively. Additionally, the article mentions Spacelift as a tool that makes working with Terraform easier by supporting Git workflows, policy as code, programmatic configuration, context sharing, drift detection, and more features.

The article discusses the issue of a Terraform state file being locked when trying to perform an operation, which can occur in team environments or shared settings due to simultaneous modifications by multiple users or processes. To prevent conflicts and potential corruption of the state file, Terraform employs a locking mechanism. However, if the lock is stuck or there are issues with the storage account holding the Terraform state file, it can be manually unlocked using the Terraform force-unlock command. The article provides detailed instructions on how to find and use the lock ID, as well as precautions when using force-unlock. It also mentions alternative methods for unlocking remote Terraform states and encourages users to explore Spacelift's features for managing Terraform state more effectively and securely.

The ImagePullBackOff error occurs when a Kubernetes Pod is unable to pull a container image required by one of its containers, causing the Pod to enter an "ImagePullBackOff" status with exponentially increasing delays between retry attempts. This error can be caused by various issues such as invalid image names, missing registry credentials, network connectivity problems, or images being removed from the registry. To troubleshoot and resolve this issue, it's essential to inspect the Pod's event history using the "describe pod" command, identify the root cause, and take corrective actions accordingly. The error is closely related to the ErrImagePull error, which occurs when an image pull fails for the first time, and can be resolved by adjusting the Pod manifest, pushing the expected image into the registry, or providing missing registry credentials. By following a methodical troubleshooting process and using the right tools, developers can solve this issue without waiting for another developer or administrator.

Continuous Integration (CI) and Continuous Delivery (CD) pipelines automate the process of taking code changes from development through to production environments, ensuring all code is subject to required checks while allowing developers to focus on their work. Docker, a popular containerization platform, can be used in conjunction with CI/CD pipelines for maximum effect. The two main ways Docker is utilized within CI/CD are: using Docker to run pipeline jobs and using a CI/CD pipeline to build and deploy Docker images. Combining these technologies helps achieve fast, simple, secure, and scalable pipelines that can improve delivery throughput while maintaining consistent quality standards.

In this blog post, we learn how to install Prometheus on Kubernetes using Helm charts. We also customize the Prometheus configuration via Helm charts to connect various node exporter instances and expose their metrics on Prometheus. The article covers key points such as the installation of Prometheus using Helm, analyzing the Prometheus dashboard, configuring Prometheus deployment, and collecting logs from non-containerized services.

The `ignore_changes` argument in Terraform is used within the lifecycle meta-argument to prevent specific attributes of a resource from being considered when determining whether to update or recreate the resource. This can effectively prevent certain changes from being applied to your infrastructure. Common use cases for ignore_changes include ignoring non-persistent or constantly changing attributes, immutable or rarely changed attributes, metadata or auxiliary information, automatically computed attributes, sensitive data, and attributes controlled by external processes. When using `ignore_changes`, it is important to ensure that the attribute names match exactly with those defined in the resource schema, and that only attributes defined by the resource type can be ignored. The argument operates at the attribute level and applies only to the resource where it's defined.

Ansible is a powerful tool that can be used to manage Kubernetes clusters. In this guide, we will cover how to use Ansible to deploy your Kubernetes environment and resources in your Kubernetes cluster. We will also discuss how you can integrate Ansible with CI/CD tools like Jenkins or GitOps tools like Argo CD for a seamless workflow. Firstly, let’s set up our Ansible Control Node. For this guide, we will assume that you have already installed Ansible on your control node and configured it to connect to your target machines (master, worker nodes, and proxy). If not, please refer to the official Ansible documentation for installation and configuration instructions. Next, let’s create an inventory file (kube_inventory) under ~/ansible/inventory/. This file will contain the IP addresses or host names of all your target machines: [master] 10.x.x.x [workers] 10.x.x.x 10.x.x.x [proxy-servers] 10.x.x.x #add your proxy IP or DNS name here. Now, let’s create a playbook file as kube_master.yml under ~/ansible/playbooks/. This playbook will be responsible for deploying the Kubernetes master node: - name: Install and Configure Kubernetes Master Node hosts: master become: yes tasks: - name: Update System Packages apt: name: "{{ item }}" state: latest update_cache: yes with_items: - software-properties-common - python3-pip - python3-venv - python3-setuptools - git - name: Install Docker CE shell: curl -fsSL https://get.docker.com/ | sudo sh args: chdir: $HOME - name: Add Kubernetes Repository Key shell: curl -s https://packages.cloud.google.com/apt/doc/repository.gpg.key | apt-key add - args: chdir: $HOME - name: Add Kubernetes Repository Line to Sources List copy: content: "deb http://apt.kubernetes.io/ kubernetes-xenial main" dest: /etc/apt/sources.list.d/kubernetes.list - name: Update System Packages apt: update_cache: yes - name: Install Kubelet, Kubeadm and Kubectl apt: name: "{{ item }}" state: latest with_items: - kubelet=1.29.2-00 - kubeadm=1.29.2-00 - kubectl=1.29.2-00 - name: Configure Kubernetes Master Node copy: content: | KUBECONFIG=/etc/kubernetes/admin.conf KUBERNETES_PROVIDER=aws dest: /home/YOUR_USERPROFILE_NAME/.bashrc owner: YOUR_USERPROFILE_NAME - name: Source Bash Profile shell: source /home/YOUR_USERPROFILE_NAME/.bashrc args: chdir: $HOME - name: Initialize Kubernetes Master Node shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> master_node_setup.log args: chdir: $HOME creates: master_node_setup.log - name: Copy Admin Conf to Proxy copy: src: /etc/kubernetes/admin.conf dest: /home/YOUR_USERPROFILE_NAME/.kube/config remote_src: yes owner: YOUR_USERPROFILE_NAME - name: Install Pod Network become: yes become_user: YOUR_USERPROFILE_NAME shell: kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml >> pod_network_setup.log args: chdir: $HOME creates: pod_network_setup.log To run the master playbook, use the following from your Ansible Control Node: ansible-playbook ~/ansible/playbook/kube_master.yml -i ~/ansible/inventory/kube_inventory Once we have successfully deployed this to the master node, we can jump into deploying the following to our worker nodes to connect our worker nodes to the master node. Let’s create kube_workers.yml under ~/ansible/playbooks/. Please make sure to replace YOUR_MASTER_IP with the IP address of your master node and also replace YOUR_USERPROFILE_NAME with the name of your user profile that is in your /home/ directory that you are installing Kubernetes under. - name: Configure Join Commands on Master Node hosts: master become: yes tasks: - name: Retrieve Join Command shell: kubeadm token create --print-join-command register: join_command_raw - name: Set Join Command set_fact: join_command: "{{ join_command_raw.stdout_lines[0] }}" - name: Join Worker Nodes hosts: workers become: yes tasks: - name: Enable TCP port 6443 (On Master) is able to connect from Worker wait_for: "host=YOUR_MASTER_IP port=6443 timeout=1" - name: Join worker to cluster shell: "{{ hostvars['YOUR_MASTER_IP'].join_command }} >> node_joined.log" args: chdir: /home/YOUR_USERPROFILE_NAME creates: node_joined.log To run the worker nodes playbook, use the following from your Ansible Control Node: ansible-playbook ~/ansible/playbook/kube_workers.yml -i ~/ansible/inventory/kube_inventory Once the playbook runs successfully, you can validate the cluster is working properly by running the commands below from your Master Node: kubectl get nodes kubectl get all -A We will now add the kube config of the master node to our /etc/kube/config of our proxy. From the master node, you can run the command below to copy the config over to your proxy: sudo scp /etc/kubernetes/admin.conf USERNAME@MASTER_NODE_IP:~/.kube/config You can validate under your ~/.kube/config in your proxy machine to make sure you view the config and also run the following from your proxy to make sure you can access your cluster from your proxy: kubectl get nodes kubectl get all -A The next step is to deploy the Kubernetes task manifest from our Ansible Control Node. But overall, you can see how much time we can save on setting up and configuring a Kubernetes cluster using Ansible. You can easily add another Ubuntu server in your Ansible inventory file and run the playbooks to add another node into your Kubernetes cluster. You have more control over the state of your Kubernetes Nodes. Before we start, validate that you are able to ping your proxy from your Ansible control node to ensure you have connectivity. Let’s modify our inventory file to include the proxy now (~/ansible/inventory/kube_inventory) that will contain the proxy IP or Host Name (if you have DNS configured). Add your proxy to your inventory as below: [master] 10.x.x.x [workers] 10.x.x.x 10.x.x.x [proxy-servers] 10.x.x.x #add your proxy IP or DNS name here. Let’s create a simple playbook file as create_namespace.yml in ~/ansible/playbooks/ as the following to create a namespace in your Kubernetes cluster: - name: Create K8S resource hosts: proxy-servers tasks: - name: Get K8S namespace kubernetes.core.k8s: name: my-namespace api_version: v1 kind: Namespace state: present Run your ansible playbook command: ansible-playbook ~/ansible/playbooks/create_namespace.yml -i ~/ansible/inventory/kube_inventory Once the playbook run is complete, go to your proxy and validate that you are able to see the namespace created by running the following: kubectl get namespace And there you have it: you have just used Ansible to deploy a Kubernetes task manifest to your Kubernetes cluster. Here are some other playbooks (Deployments, Services, and Configmaps) you can test running from your Ansible Control Node. You can utilize the following Application/Service Deployment task manifest to deploy a nginx application: - name: Application Deployment hosts: proxy_servers tasks: - name: Create a Deployment kubernetes.core.k8s: definition: apiVersion: apps/v1 kind: Deployment metadata: name: myapp namespace: my-namespace spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: myapp-container image: nginx:latest ports: - containerPort: 80 - name: Expose Deployment as a Service kubernetes.core.k8s: definition: apiVersion: v1 kind: Service metadata: name: myapp-service namespace: my-namespace spec: selector: app: myapp ports: - protocol: TCP port: 80 targetPort: 80 type: LoadBalancer You can also manage your Kubernetes environment variables with Ansible using a configmap: - name: Manage ConfigMaps and Secrets hosts: proxy_servers tasks: - name: Create ConfigMap kubernetes.core.k8s: definition: apiVersion: v1 kind: ConfigMap metadata: name: app-configmap namespace: my-namespace data: config.json: | { "key": "value" } - name: Create Secret kubernetes.core.k8s: definition: apiVersion: v1 kind: Secret metadata: name: myapp-secret namespace: my-namespace stringData: password: mypassword Nowadays, we mostly navigate towards a PaaS solution for our Kubernetes cluster, as it will be hosted by Azure, AWS, GCP, or others. I want to briefly cover how you would connect an Azure AKS cluster to your Ansible – proxy workflow. The same process will follow for Amazon EKS and Google GKE with their dedicated CLI commands. Let’s go to our Proxy machine and run the code below. We will install the Azure CLI tools and use ‘az login’ to log in to Azure. This will validate that we are able to connect to our AKS cluster from our proxy and make sure that we have the kube config in our proxy updated. #Install Azure CLI (or any other cloud provider CLI tools): curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash #Login to Azure: az login #Add Azure AKS cluster to proxy ~/.kube/config az aks get-credentials --name name_of_aks_cluster --resource-group name_of_aks_rg #Test access to K8S cluster: kubectl get nodes kubectl get all -A Once we validate we can access our AKS cluster nodes and other resources, we can move to the Ansible Control Node and run some of the previous playbooks against the proxy. - name: Create K8S resource hosts: proxy-servers tasks: - name: Get K8S namespace kubernetes.core.k8s: name: my-namespace api_version: v1 kind: Namespace state: present Run your ansible playbook command: ansible-playbook ~/ansible/playbooks/create_namespace.yml -i ~/ansible/inventory/kube_inventory Once the playbook run is complete, go to your proxy and validate you are able to see the namespace created by running the following: kubectl get namespace We have confirmed we can run playbooks against our Azure AKS cluster. One thing to note is that we replaced the existing /.kube/config with the Azure AKS cluster config. Typically, you will have a multi-cluster environment and will need to add different config files in your ~/.kube/ location and configure your Ansible playbooks to point to the correct config file using the following: - name: Set Kubernetes context k8s_auth: kubeconfig: /path/to/kubeconfig register: kube_auth Implementing Ansible in your CI/CD workflows consists of two main methods. - Use Ansible in a Jenkins Pipeline setup for CI/CD, which allows for direct deployment and configuration of Kubernetes resources from within the pipeline. Jenkins can trigger Ansible playbooks as part of the deployment process and apply the changes directly to the Kubernetes cluster. This is an ideal approach if you are looking for a more hands-on, scriptable method to manage Kubernetes deployments. - Integrate Ansible with a CI/CD GitOps tool for Kubernetes, such as Argo CD or Flux, as these tools will focus more on the pre-processing steps that are needed to generate the Kubernetes manifests before deployment. Since Argo CD/Flux focuses on reading the Git Repository for Kubernetes manifest file changes, you can add a step in your CI/CD pipeline to trigger an Ansible playbook to dynamically generate or update the manifest files in the repository based on configurations and environments through jinja2 templates. Ansible’s strength is that it can handle all idempotent operations, which ensures consistent deployments without unnecessary reconfigurations. Ansible in a Jenkins CI/CD Pipeline Here is an example of how you would use Ansible to deploy Kubernetes manifest in a Jenkins CI/CD Pipeline: Jenkins file: pipeline { agent any environment { ANSIBLE_HOST_KEY_CHECKING = "False" } stages { stage('Checkout') { steps { checkout scm } } stage('('('('('('('('('('('('('('('('('('('('('('('('('('('('('('('

DevOps, which has been around for about 15 years, is a methodology that drives organizations to more efficient, faster, and robust delivery processes by bridging the gap between development and operations teams. Two main frameworks in the DevOps landscape are Three Ways and CALMS. DevOps is not limited to large enterprises; it can be beneficial for startups as well. The role of DevOps in a startup should focus on establishing an effective delivery chain, measuring progress, configuring and managing cloud resources, enabling infrastructure as code (IaC) and continuous integration/continuous deployment (CI/CD) systems, and fostering collaboration between people. Key benefits for startups include improved collaboration, delivery, established standards, and measurements.

Kubernetes Jobs are used to execute one-off tasks in your cluster by automatically creating Pods and managing their lifecycles until they terminate successfully. They enable scenarios where you need to run extra tasks separately from your main application, such as background database migrations or work queues. Jobs can be configured for non-parallel processes, multiple tasks in parallel (work queue), and multiple tasks in parallel (fixed completion count). They are created from YAML manifest files and can be controlled for parallelism and failure handling. Kubernetes CronJobs allow you to start Jobs on a recurring schedule using cron syntax.

GitLab CI/CD is a continuous integration and delivery system that can be used to manage Terraform state files. It provides features such as pipeline configuration, runner distribution, job execution, artifact storage, and environment variable management. Using GitLab CI/CD for managing Terraform state offers benefits like version control, centralized repository, encryption, access control, collaboration, locking and unlocking states, and remote execution of commands. Spacelift is another platform that can manage Terraform state files with additional features such as OPA policy integration, self-service infrastructure, drift detection, and workflow automation.

Pulumi is an open-source Infrastructure as Code (IaC) tool that allows developers to define infrastructure components in various programming languages, including Javascript, Python, Typescript, Go, C#, and Java. It streamlines infrastructure management with features like modular configuration, secrets management, and automation. The state backend is crucial for IaC as it stores metadata about the desired configuration and what has been provisioned in the real world. State backends can be classified into service and self-managed types, with local and 3rd party state backends being further classifications of the latter. This post provides a hands-on walkthrough of configuring a local backend, creating an EC2 instance using Pulumi and Typescript, migrating the state to Pulumi Cloud, and deleting/destroying the EC2 instance. Spacelift is also mentioned as a flexible management platform for IaC that supports various tools like Pulumi, Terraform, Kubernetes, CloudFormation, Ansible, and Terragrunt.

The rapid growth of cloud infrastructure management has led to an increased demand for DevOps, cloud, and site reliability engineers. Infrastructure as Code (IaC) tools like Terraform have become popular for managing cloud infrastructure using a declarative language called HCL. However, there are limitations with Terraform that can be addressed by HashiCorp's managed offering, Terraform Cloud. This service offers features such as state management, secrets, policy as code, integrations with third-party tools, credentials storage, and role-based access controls to streamline the infrastructure deployment process. Despite its benefits, Terraform Cloud has limitations like vendor lock-in, pricing based on resources under management (RUM), and limited support for other IaC workflows. Some alternatives to Terraform Cloud include Spacelift, Pulumi Cloud, GitHub Actions, GitLab CI/CD, CircleCI, Env0, Scalr, and Atlantis. Among these options, Spacelift stands out as a dedicated platform for IaC management that enhances governance and security while simplifying the overall deployment process of infrastructure.

Jenkins, once a pioneer in the CI/CD space, now faces competition from modern alternatives that offer improved VCS integration, support for cloud-native deployments, and compatibility with IaC and GitOps workflows. This article explores ten popular Jenkins alternatives, including Spacelift, GitHub Actions, GitLab CI/CD, CircleCI, Travis CI, CodeShip, AWS CodePipeline, Azure Pipelines, Bitbucket Pipelines, and TeamCity. These alternatives offer simpler, faster, and more cloud-native solutions that can better adapt to contemporary software delivery methods.

Checkov is a static code analysis tool that scans for security vulnerabilities in Infrastructure as Code (IaC) configurations, providing greater assurance in the security and reliability of infrastructure deployments. It supports multiple IaC tools, including Terraform, CloudFormation, Kubernetes YAML files, and Serverless Frameworks. Checkov can be integrated into CI/CD pipelines to detect misconfigurations and vulnerabilities early in the development lifecycle. The tool also provides a wide range of built-in checks that evaluate the configuration files against security best practices and compliance standards. Custom policies can be defined to enforce internal policies and best practices beyond the built-in checks or exclude certain checks as required. Checkov supports multi-cloud environments, has extensive rule coverage, is customizable, integrates with CI/CD tools, supports multiple reporting formats, and has an active community and rule contributions. To run a Checkov scan, you need to install it and then configure an input folder containing your files or a single file for Checkov to target. The tool can be used with Azure DevOps pipelines to verify modules and root files, as well as with Terraform to validate configuration files and plans. Additionally, Checkov supports Kubernetes, Helm, and Kustomize configurations. It can detect potential misconfigurations in IaC, such as overly permissive security group rules or weak encryption settings, and provides recommendations for remediation. By integrating Checkov into your CI/CD pipeline, you can improve the security, reliability, and compliance of your infrastructure deployments.