The cost of knowledge

The text explores the challenges of managing high cardinality in observability systems, emphasizing the trade-offs between cost and knowledge. Cardinality, which refers to the number of unique combinations of labels (or dimensions) in a data set, is often seen as a burden due to the storage and query costs associated with it. High cardinality is inevitable in modern infrastructures like Kubernetes, where frequent changes lead to identity churn. This complexity is compounded by business dimensions, causing a dramatic increase in data points that need to be processed. The text discusses the implications of dropping labels to reduce costs, such as losing critical insights and potentially corrupting data. It argues that while traditional SaaS and DIY systems impose a cardinality tax or operational strain, some platforms focus on managing ingestion volume to avoid forcing users to trade knowledge for budgetary reasons. The overarching message is that reducing cardinality might save money in the short term, but it can also lead to a loss of valuable insights when they are most needed.