August 2023 Summaries

The managed Kubernetes solutions from major cloud providers like AWS, Google Cloud Platform, and Azure usually have capabilities to autoscale GPU node pools. However, Kubernetes GPU autoscaling quickly gets tricky as you have to configure GPU node pools manually, which may lead to nodes lingering for a while, increasing your cluster costs. CAST AI's autoscaling and bin packing engine provisions GPU instances on demand and downscales them when needed, taking advantage of spot instances and their price benefits to drive costs down further. Currently, CAST AI supports GPU workloads on Amazon Elastic Kubernetes Service (EKS) and Google Kubernetes Engine (GKE), with support for Azure Kubernetes Service (AKS) coming soon.

Kubernetes taints and tolerations are used to control pod scheduling on nodes, allowing administrators to reject certain pods from tainted nodes. Taints are applied to nodes to repel specific pods, while tolerations are applied to pods to allow them to schedule on tainted nodes. This combination provides granular control over pod scheduling and can be used to dedicate nodes for specific workloads or applications. It also helps in cost optimization by ensuring that only necessary resources are allocated to pods. Additionally, taints and tolerations can be used to automate the lifecycle of spot instances, which offer excess capacity at a large discount but come with the risk of interruption. By using Kubernetes taints and tolerations, administrators can improve the efficiency and reliability of their cluster while reducing costs.

Kubernetes labels are key-value string pairs that link identifying metadata to Kubernetes objects. They provide teams with integrated support for using labels to retrieve and filter the data from the Kubernetes API and carry out bulk operations on selected objects. Labels can be used to group resources, perform bulk operations, schedule pods based on node labels, and aid in cost monitoring. To effectively use Kubernetes labels, it is recommended to follow best practices such as standardizing label naming conventions, avoiding unnecessary changes to labels, using label-selection options, not storing application-level semantics or sensitive information in labels, adding labels to pod templates, automating the labeling practice, and utilizing labels for cost monitoring. By implementing these best practices, teams can realize tremendous benefits from infrastructure visibility and efficient operations.

Microsoft's Active Directory (AD) is a directory service for Windows domain networks that has become an umbrella term for diverse identity-related services. AD enables authentication and authorization, making it a crucial component of enterprise security. However, its on-premise version can be complex to manage, especially in large organizations with multiple users and permissions. A cloud-based version of Active Directory can simplify identity management by providing a centralized identity source and enhancing security through unified authentication and authorization. The three main AD-based solutions are Active Directory Domain Services (AD DS), Azure Active Directory (Azure AD), and Azure Active Directory Domain Services (Azure AD DS). Choosing the right version depends on the organization's specific needs, such as the level of control required over directory management, the need for legacy authentication services, and the complexity of the cloud deployment. A managed PaaS offering for AD DS can provide a convenient solution for organizations that don't require full control over directory management. Ultimately, selecting the right Active Directory solution is crucial for a smooth cloud migration and ensuring consistency with existing on-prem systems.

The article highlights the often-overlooked issue of data egress costs in cloud adoption. Research from IDC shows that these charges account for an average of 6% of organizations' cloud storage costs, and can be unpredictable, making it challenging to forecast and model them. Data egress fees are charged whenever data is moved out of a cloud storage location, and can add up quickly as an organization's cloud footprint grows. This can lead to vendor lock-in, limit the ability to fully benefit from the cloud, and raise operating expenses for hybrid and multi-cloud architectures. To reduce egress fees, companies can adopt demand management, architect for lower costs by prioritizing reduced inter-regional data transfers, use of data deduplication and compression, and redesigning data-intensive apps, choose the right services, bring data on-premises, and consider solutions that eliminate egress costs, such as CAST AI.

Using in-tree storage drivers can lead to support being dropped for Kubernetes v1.26 and should be replaced with Container Storage Interface (CSI) to ensure compatibility and future-proofing. Lack of Pod Disruption Budgets can cause applications to become unavailable due to evictions, while unrealistic budgets that prevent any pod movement can block node lifecycle activities like patching or bin packing low-utilization nodes. Isolating workloads without strong requirements for it can lead to performance and availability issues, as well as cost inefficiencies. Mixing competing placement constructs such as NodeAffinity and NodeSelectors can cause configuration errors and inconsistencies. Not having observability in place can make troubleshooting difficult, while using burstable instances to reduce cloud costs can often be more expensive than expected. Excessive cross-AZ traffic can increase cloud bills unexpectedly. Not setting resource requests for applications can lead to CPU throttling and OOM kills. Failing to expose application health status through Liveness and Readiness probes can make it difficult to diagnose issues. Scaling workload replicas manually can be wasteful and fragile, while running a self-hosted Kubernetes control plane can be unnecessary and costly. Not using namespaces or keeping everything in the "default" namespace can lead to performance and maintenance issues. Failing to add wildcardable tolerations to vital DaemonSets can cause blind spots in logging coverage. Running business logic in DaemonSets is often not necessary and can lead to outages. Applications unable to terminate gracefully in a short time window can create cascading issues and increase mental overhead.

Spot instance prices have been surging recently, with Azure leading the way with a 108% increase from 2022 to 2023, followed by AWS at 21%. GCP, however, has seen a nearly 26% decrease in spot VM prices. The price increases vary across regions and instance types for each cloud provider. By being flexible and using automatic provisioning tools like CAST AI's scheduled rebalancing feature, users can navigate these changes and save significantly on their cloud spend.

CAST AI introduces scheduled rebalancing, a new feature that helps users protect their cloud savings from spot instance pricing changes. Spot instances can be cost-efficient but are subject to price hikes due to market supply and demand. The scheduled rebalancer allows users to set criteria for when more cost-effective spot instances become available, mitigating the negative effects of price hikes. This feature enables teams to create cloud cost optimization plans that cover different use cases and automatically execute them based on preferences. CAST AI clients save an average of 63% on their Kubernetes bills with this technology.

Azure offers a range of container-as-a-service options for teams moving traditional monolithic software to the cloud and containers, simplifying application deployment and maintenance. AKS is a fully managed Kubernetes service that provides a standards-based Kubernetes implementation and integrates well with other Azure services. ACI offers a way to run container workloads in Azure without worrying about managing the underlying infrastructure. Azure Service Fabric is a container orchestrator that can deploy and manage microservices across a cluster of machines, providing robust support for stateful services via containerized stateful services or built-in Service Fabric programming models. Azure Batch + Batch Shipyard allows you to run long-running batch jobs on dedicated compute capacity, while Azure Red Hat OpenShift provides a fully-managed variant of the OpenShift platform, offering storage management, logging and monitoring, and image registries to extend Kubernetes. Azure Functions is a cloud service that lets developers run code continuously, providing serverless compute for Azure, and Azure Container Apps let you run microservices and containerized applications on a serverless platform. The choice of service depends on the specific requirements of your application, including factors such as complexity, scalability, and cost.

Nvidia is diverting its AI chip supply away from large cloud providers like AWS, instead targeting smaller ones, while cloud giants are developing their own AI chips to reduce reliance on Nvidia. Microsoft has made cloud security logs free for all customers as a precautionary measure following a reported China-based hack of US government officials' cloud-based emails. The cloud industry is witnessing a shift towards larger-scale AI applications due to the growing need for more powerful infrastructure, but it's uncertain how this will impact existing cloud services and pricing models.