September 2022 Summaries

Event streaming is becoming increasingly crucial for modern businesses, with Apache Kafka being a popular open-source solution for managing real-time data flows between producers and consumers. Two primary managed Apache Kafka service providers, Confluent and Amazon MSK, offer different pricing models and features. Amazon MSK, particularly its serverless option, is often the more economical choice for businesses with predictable workloads and in-house Kafka expertise, due to its lower server costs. Conversely, Confluent, despite being generally more expensive, provides a comprehensive data streaming platform with advanced features such as ksqlDB, managed connectors, and options for deployment beyond AWS, making it suitable for companies needing robust support and feature sets. Confluent also offers a free tier, unlike Amazon MSK, which can be beneficial for businesses evaluating their options. Ultimately, the decision between the two depends on the specific needs, expertise, and workload predictability of the business.

Vantage has introduced a new feature that automatically detects and alerts users to cost anomalies across various providers like AWS, GCP, Datadog, Snowflake, and Fastly, aiming to prevent unexpected expenses by notifying users when costs deviate from typical patterns. This cost anomaly detection is enabled by default for all users and can be accessed in the "Anomalies" section of cost reports, allowing customers to configure alerts via Slack, email, or Microsoft Teams. The system uses a machine learning model to identify significant cost increases while filtering out low-value deviations to reduce unnecessary alerts. Anomalies have three states—Active, Archived, and Ignored—and alerts are sent only once when a new anomaly is detected, with the option for users to provide feedback on false positives. Although currently unable to alert specific resources or based on existing budgets, Vantage plans to expand these capabilities, and users interested in future developments like Kubernetes-specific anomaly detection can join beta programs for early access.

AWS has introduced Graviton2 processors for Lambda users, significantly reducing costs and influencing a shift from x86 to ARM architecture among customers. Despite AWS's reputation for high cloud costs, they aim to help customers save money, with Vantage data suggesting that ARM will soon dominate Lambda costs. The pricing structure for Lambda involves CPU runtime, memory consumption, and request frequency, with ARM offering a 34% lower cost than x86. Migrating Lambda functions to ARM varies in difficulty depending on the runtime and presence of binary dependencies, with AWS benefiting from better margins despite offering lower customer costs. This shift aligns customer savings with AWS's financial interests, as running Lambdas on Graviton2 is more efficient and cost-effective.

AWS users can inadvertently waste money by over-committing to Reserved Instances (RIs) or Savings Plans, resulting in unused contracts and financial inefficiencies. To address these issues, various strategies can be employed, such as selling RIs on the AWS RI Marketplace, converting RIs to other types or into Savings Plans, and managing the liquidity of specific instance types. Selling RIs typically requires a US bank account, but alternatives like using Hyperwallet, a PayPal service, can be explored. While Savings Plans offer substantial discounts, they lack a resale market, making it crucial to assess infrastructure needs carefully before committing. Combining RIs and Savings Plans can optimize coverage, though it requires careful management. Ultimately, these approaches can mitigate the financial impact of over-commitments, but proactive and strategic purchasing remains the best practice. Vantage provides tools and consultation to help manage AWS discounts effectively.

The AWS Cost and Usage Report (CUR) offers the most detailed billing data available, providing insights into service cost and consumption, with reports that can reach sizes of tens to hundreds of gigabytes. To effectively work with CUR data, users must understand its complex schema, which includes tables such as identity, bill, lineItem, pricing, and product, among others that populate under specific conditions like savings plans or tagged resources. CUR data must be enabled in the AWS billing console and stored in an S3 bucket, after which it becomes available for daily updates, though it is not finalized until the end of the month. Users can parse CUR data using various methods, including opening CSV files in Excel or using programming languages to extract useful information. The report is essential for understanding billing details, managing costs, and identifying savings opportunities, while integrations with tools like Athena, Quicksight, and Redshift enable deeper data queries for cost visibility and optimization.