June 2026 Summaries

Karpenter is an open-source autoscaler designed for Kubernetes that efficiently provisions EC2 instances directly through the AWS API, bypassing the need for Auto Scaling Groups, and offers significantly faster node provisioning compared to the traditional Cluster Autoscaler. With Karpenter, nodes can be ready in 45–60 seconds under typical conditions, while Cluster Autoscaler takes 3–4 minutes. The current stable release, Karpenter v1.13.0, introduces a new API, replacing older components with NodePool and EC2NodeClass for improved management. The guide provides a detailed walkthrough for deploying Karpenter on EKS, highlighting prerequisites such as Kubernetes 1.25 or later, AWS CLI v2, and Helm 3. It also emphasizes the benefits of Karpenter's per-pod instance type selection from the entire EC2 catalog and its native support for AWS features like Spot instances. While Karpenter excels in speed and resource management, it relies on pod resource requests for provisioning, which can lead to overprovisioning if requests aren't optimized. To address this, Cast AI offers an additional optimization layer called KENT, which adjusts resource requests based on actual usage and provides cost visibility. The guide also advises against running Karpenter and Cluster Autoscaler simultaneously to avoid conflicts and recommends specific configurations to ensure smooth operation and cost efficiency.

Karpenter is an open-source node provisioner that offers a direct API request approach for provisioning cloud instances, optimizing instance selection at scheduling time, and allowing for faster node readiness and greater instance flexibility compared to the Cluster Autoscaler (CAS), which operates within pre-defined node groups and typically takes longer to provision nodes. In 2026, Karpenter is the preferred choice for AWS and Azure due to its efficient node consolidation and native support for Spot instances with automatic fallback, while CAS remains the only viable option for GKE users as no official Karpenter provider exists for GCP. While Karpenter can consolidate workloads onto fewer nodes effectively, neither Karpenter nor CAS address pod-level overprovisioning, which is significant for optimizing overall costs. The addition of Cast AI's optimization stack, which includes pod-level rightsizing, can significantly enhance cost savings when used alongside Karpenter. Running both Karpenter and CAS simultaneously is possible but requires careful partitioning to avoid conflicts during migration.

GPU optimization tools aim to enhance the efficiency of GPU usage by improving the cost-to-output ratio through various methods such as node lifecycle automation, partition management, workload sharing, Spot orchestration, and cost attribution. The concept of a GPU Cost Optimization Loop, consisting of Measure, Allocate, Share, and Automate phases, underscores the necessity of multiple tools working cohesively for effective optimization. Despite the potential for significant savings, the underutilization of GPUs, especially when not leveraging Spot instances, presents a major cost inefficiency. Tools like NVIDIA's Multi-Instance GPU (MIG) and time-slicing provide means to maximize GPU resource utilization, each with specific use cases depending on workload requirements. The integration of these tools into a synchronized system, such as that offered by Cast AI, is vital for managing resources across multi-cloud environments effectively. The economic landscape is increasingly challenging, with rising GPU costs and low utilization, prompting organizations to adopt autonomous optimization strategies to mitigate financial waste.

Kubernetes cost management and optimization tools are essential for monitoring, allocating, and reducing the expenses associated with running workloads on Kubernetes. Cost management focuses on visibility by tracking and reporting spend across clusters, namespaces, and workloads, while cost optimization emphasizes actionable strategies like rightsizing pods, autoscaling nodes, and using Spot Instances to lower bills. The 2026 State of Kubernetes Optimization Report highlights significant waste, such as low CPU and GPU utilization and high overprovisioning rates, suggesting the need for both visibility and autonomous optimization platforms. Tools like Cast AI offer comprehensive solutions by automating cost reductions across all waste layers, while others like Kubecost and OpenCost focus on providing detailed cost visibility and reporting. Organizations are encouraged to identify their primary sources of waste to choose the most appropriate tool, whether for visibility, optimization, or both, to achieve substantial cost savings and improved resource efficiency.

CrashLoopBackOff is a Kubernetes pod status indicating that a container repeatedly starts, crashes, and is restarted by the kubelet, with exponential backoff between attempts to avoid overloading the node. This status is not an error code but a waiting state due to issues such as application configuration errors, out-of-memory (OOM) kills, failing liveness probes, incorrect images or entrypoints, missing dependencies, or init container failures. The backoff sequence begins at 10 seconds, doubling with each failure up to a cap of 5 minutes, resetting after 10 minutes of successful operation. Diagnosing these issues involves a structured process of checking pod status, reviewing logs, and examining events to pinpoint the underlying cause. Solutions include adjusting memory limits, fixing configuration errors, and implementing startup probes for slow-starting applications. Tools like Cast AI Workload Autoscaler can automate memory limit adjustments to prevent OOM-driven crash loops, while OpsPilot aids in rapid diagnosis by identifying root causes and restart counts quickly.

OOMKilled in Kubernetes refers to the termination of a container by the Linux kernel's Out-Of-Memory (OOM) killer due to exceeding its cgroup memory limit, resulting in exit code 137. This occurrence is reported by Kubernetes but is enforced by the Linux kernel. The scenarios leading to OOMKilled include a container exceeding its individual memory limit, node memory pressure causing kubelet eviction, and node-level kernel OOM events. Diagnosing and addressing OOMKilled involves inspecting container states and logs, setting appropriate memory requests and limits based on actual usage, and ensuring sufficient node memory headroom. While raising memory limits might offer a temporary solution, accurate per-container sizing and automated rightsizing are recommended for long-term prevention. Tools like Cast AI Workload Optimization can proactively adjust memory limits based on real-time data, reducing the frequency of OOM kills and optimizing resource allocation across clusters.

The document provides a detailed exploration of exit codes in Kubernetes, focusing on the significance and troubleshooting of common exit codes like 137, 139, and 143, which correspond to process terminations due to out-of-memory conditions, segmentation faults, and SIGTERM signals, respectively. It explains how these codes are derived from POSIX standards and how Kubernetes surfaces them in pod status, emphasizing that an exit code above 128 indicates a process killed by an OS signal. The text also outlines the Exit Code Diagnostic Loop as a method for identifying and resolving issues, recommending steps to classify, fix, and prevent crashes by examining logs, events, and pod specifications. Additionally, it discusses the role of resource configurations in causing exit codes, particularly the structural nature of exit code 137 due to static memory limits, and suggests using tools like Kubernetes Vertical Pod Autoscaler or Cast AI’s Workload Autoscaler for dynamic resource allocation. The importance of handling signals correctly, especially with proper shutdown procedures and entrypoint configurations, is highlighted to avoid unexpected crashes and ensure smooth operations.

In the context of choosing between TPUs and GPUs for machine learning infrastructure, the decision primarily hinges on silicon architecture, workload compatibility, and total cost of ownership. TPUs, with their Matrix Multiply Units, excel in large-scale pretraining with JAX or MaxText on GCP, particularly for workloads with static input shapes and large batch sizes, offering significant performance in specific scenarios like sparse embedding table lookups. They are best suited for organizations already embedded within the JAX ecosystem, such as Google and DeepMind, due to their high raw throughput and cost-effectiveness with long-term committed use. Conversely, GPUs, particularly NVIDIA's offerings, dominate the broader market due to their flexibility, mature CUDA ecosystem, and capability to handle dynamic input shapes, making them the default choice for most production ML models, including those using PyTorch. GPUs are more versatile, supporting a wider range of tools and frameworks across multiple cloud providers, and are often more cost-effective when utilizing spot instances for dynamic workloads. The framework migration cost and the necessity for static shape discipline with TPUs often tip the balance in favor of GPUs for teams prioritizing flexibility and multi-cloud compatibility.

Karpenter is a tool designed to optimize the scaling of EKS clusters, but deploying it in production environments requires careful configuration to avoid common pitfalls. The stable v1 API, released in mid-2024 and maintained under the kubernetes-sigs GitHub organization, introduces improvements such as NodePool and EC2NodeClass, disruption budgets, and an SQS interruption pipeline to manage node disruptions effectively. Best practices for using Karpenter include running it on dedicated nodes to avoid circular dependencies, designing mutually exclusive NodePools to prevent unpredictable workloads, and setting resource limits to avoid over-provisioning. Spot interruption handling should be managed via SQS for better response times, and AMIs should be pinned to prevent untested versions from disrupting operations. Additionally, maximizing instance-type diversity for Spot, configuring consolidation and disruption budgets, setting node expiry for AMI freshness, and using the do-not-disrupt annotation strategically can enhance Karpenter's effectiveness. Monitoring with Prometheus is essential for identifying bottlenecks and disruptions, and integrating Cast AI can further optimize resource requests and Spot utilization. These practices ensure that Karpenter scales clusters reliably and safely, offering a more predictable and efficient infrastructure.

The benchmark study compared four different Kubernetes cluster consolidation strategies using the Karpenter provisioner and Cast AI tools, demonstrating varying cost efficiencies over a seven-day period with adversarial workload conditions on AWS EKS. The baseline approach solely relied on Karpenter's built-in consolidation policy, which was found to be limited due to its inability to perform coordinated multi-node reshuffling, resulting in the highest cost of $703.08. A second variant integrated Cast AI's Evictor, a continuous bin-packing daemon, leading to modest savings of 9.1% by continuously evicting underutilized nodes. The third variant, which employed Cast AI's Continuous Rebalancer from the Karpenter Enterprise Suite, improved savings to 15.8% by enabling more targeted instance selection and coordinated node drains. The most cost-effective solution, the Cast AI Autoscaler, replaced Karpenter entirely, achieving a 43.0% reduction in costs by optimizing workload fit during provisioning, thereby minimizing the need for post-hoc corrections. This study underscores the potential for significant cost reductions through strategic consolidation approaches, particularly when provisioning decisions are optimized from the outset.

At the FinOps X 2026 conference, the focus was on the evolving role of FinOps in managing AI infrastructure costs, particularly in GPU inference, which has become a significant expense for organizations. The discussion highlighted the transformative potential of AI in enhancing developer productivity, allowing junior roles to perform at senior levels, and emphasized the need for companies to continue hiring entry-level positions to sustain career growth. A key point was the importance of selecting the appropriate AI models for specific tasks to avoid unnecessary expenses, as not all models are equal in terms of cost and output quality. The solution proposed involves using AI to automate the selection of models, optimizing costs and maintaining performance. This approach mirrors Cast AI's strategy for Kubernetes infrastructure, where autonomous agents handle resource allocation and cost management. Furthermore, the introduction of the Tokenomics Foundation was announced, marking a shift in focus towards AI consumption economics, positioning FinOps practitioners as crucial in navigating this landscape. Laurent Gil, Co-founder of Cast AI, advocates for continuous optimization and automation in AI infrastructure to ensure efficiency and cost-effectiveness.

Platform engineers face challenges in maintaining stable production environments due to the uniform application of provisioning logic across different environments, leading to overprovisioning in Kubernetes clusters. The 2026 State of Kubernetes Optimization Report highlights widespread overprovisioning, with CPU and memory utilization averaging 8% and 20%, respectively. While overprovisioning in production environments is somewhat acceptable to ensure stability, it results in waste in development and staging environments where workloads run intermittently. Traditional tools lack the flexibility to enforce differentiated rightsizing strategies, and manual adjustments are cumbersome. Cast AI's OpsPilot addresses this by using AI to interpret cluster signals and generate custom policies based on real-time data, workload behavior, and risk profiles, allowing for tailored resource management. OpsPilot automates policy creation and adjustment, eliminating the need for static rules and manual interventions, and is available for all Cast AI customers, offering continuous, AI-driven optimization of Kubernetes workloads.

At the FinOps X conference in San Diego, J.R. Storment announced the launch of the Tokenomics Foundation, marking a significant evolution in the FinOps discipline by focusing on the efficient governance of AI token consumption and linking it to business value. The event emphasized the exponential growth of AI token usage, projected to reach 120 quadrillion tokens in three years, and highlighted the need for infrastructure teams to address this challenge by optimizing GPU usage, employing spot instances, and ensuring autoscaling and multi-cloud capacity routing. Speakers emphasized that understanding the full cost of AI workloads goes beyond invoice analysis, requiring a comprehensive approach to infrastructure and model selection that can autonomously manage resources and costs in real-time. The introduction of the Tokenomics Foundation aims to place FinOps practitioners at the center of strategic discussions, ensuring organizations are prepared to define the unit economics that will shape the future of AI infrastructure management.