December 2023 Summaries

Danielle Vansia's blog post provides a comprehensive analysis of Snowflake's compute costs, which typically represent a significant portion of the overall bill, and suggests various optimization strategies for managing these expenses. The compute costs are divided into three main categories: virtual warehouse compute, serverless compute, and cloud services compute, with each category having specific billing criteria and optimization opportunities. Virtual warehouses, which can be managed for size and uptime, are billed based on a consumption model where charges apply only when resources are active, while serverless services are based on compute-hours for Snowflake-managed tasks. Cloud services costs are calculated separately and only billed when they exceed a certain percentage of virtual warehouse usage, with adjustments made daily. The article emphasizes the importance of selecting the appropriate virtual warehouse size, setting timeouts, leveraging query caching, and being judicious with serverless features to reduce unnecessary expenses. Additionally, it highlights the need for careful management of cloud services resources to prevent excessive charges, advocating for strategies such as avoiding full database clones and monitoring complex queries. Overall, the post aims to equip Snowflake users with the knowledge to better understand and optimize their compute spending, thereby avoiding unexpected charges and improving cost efficiency.

Cost vs usage is a critical concept in building cost reports for different departments, particularly in FinOps, where detective work is vital for understanding unexpected cost fluctuations. In a scenario where a company's engineering team saw a 20% cost increase without any product changes, the investigation revealed that the issue stemmed from a shift in Savings Plan coverage, favoring another team's new product and inadvertently increasing costs for the first team. This highlights the importance of tailoring cost reports to different needs; while finance departments benefit from reports that match invoice totals, engineering teams require reports that reflect cost changes resulting from their actions, filtering out external factors like discounts. Effective cost visibility programs should focus on providing meaningful and actionable insights to the relevant teams, ensuring clarity and accountability in managing cloud expenses.

Amazon Web Services (AWS) has introduced the Infrequent Access log class to help reduce CloudWatch Log costs, which are a significant portion of AWS expenses, with logs accounting for 43% of costs according to Vantage Cloud Cost Reports. This log class offers up to 50% savings on ingestion costs by reducing available features, making it suitable for logs that are not frequently accessed, such as debug logs, web server logs, and compliance logs. It is particularly beneficial for scenarios where advanced features are unnecessary, providing a cost-effective solution without sacrificing necessary functionality. The Infrequent Access log class can be particularly advantageous for workloads where data is accessed rarely or requires only ad-hoc querying, as demonstrated by case studies involving financial companies and software development teams, both of which achieved significant cost savings.

Vantage has introduced Kubernetes Efficiency Reports to help teams identify inefficiencies in Kubernetes pods by providing customized views of idle and total costs based on Clusters, Namespaces, and Labels. These reports enable users to create comprehensive dashboards that display total and idle costs across different providers, including Kubernetes, and allow the integration of these reports with other cost and segment reports. Users can save and customize reports with filters, aggregations, and date ranges, and choose whether to display idle or total costs by default. While Kubernetes total cost data is available via the API, idle costs are not yet supported but may be added in the future. This feature is designed for customers managing shared Kubernetes clusters who need insights into cost contributions from different teams or services.

Amazon S3 Express One Zone, introduced at the 2023 AWS re:Invent conference, is a new storage class designed for latency-sensitive applications, offering ten times better performance than S3 Standard and providing consistent single-digit millisecond latency. It operates within a single Availability Zone (AZ) and is particularly suitable for frequently accessed data or demanding applications, with a capacity to handle hundreds of thousands of transactions per second. Despite its high performance, S3 Express One Zone's storage costs are significantly higher—approximately seven times—than those of S3 Standard, though request costs are reduced by 50%. The community response to the new storage class is mixed, with some users appreciating the performance improvements and others concerned about its cost implications, especially given the need for manually replicating data to avoid single-AZ failures. Overall, while S3 Express One Zone offers notable latency improvements, its higher costs may deter some users, and its adoption will depend on specific use cases and data access patterns.

Vantage has introduced new API endpoints that allow users to manage cost allocation segments, enabling them to list, create, update, and delete segments programmatically. Previously, segment management was confined to the Vantage console, limiting automation and flexibility for users with complex segment structures. With the new API capabilities, users can automate the creation and modification of segments, facilitating easier integration of hierarchical data from sources like spreadsheets. The API uses the Vantage Query Language for filters, and the feature is available to all customers with the segments feature enabled, free of charge. Although real-time updates to cost data are not yet possible, the API provides a more efficient means of segment management, with further enhancements planned on the roadmap.

Incorporating the Terraform Cloud Development Kit (CDKTF) into a FinOps as Code strategy enables teams to manage cloud costs more efficiently by utilizing familiar programming languages like TypeScript, Python, Java, C#, and Go. CDKTF, built on the AWS Cloud Development Kit, translates code into Terraform-compatible configurations, offering more flexibility and higher-level abstractions compared to the traditional HashiCorp Configuration Language (HCL). This approach supports modularity and reusability, allowing developers to create reusable infrastructure templates while optimizing costs. CDKTF is particularly advantageous for teams with existing programming experience, as it allows the use of constructs to abstract and reuse infrastructure components, aligning with FinOps principles like timely reporting and collaboration. By providing a method to create scalable and reusable cost reporting infrastructures with tools like Vantage, CDKTF fosters a collaborative environment for cloud cost optimization, promoting a seamless integration of FinOps practices into existing workflows.