June 2020 Summaries

The text discusses the considerations and trade-offs involved in choosing open-source software (OSS) for data warehousing technologies, challenging common assumptions that OSS is cheaper, avoids vendor lock-in, and offers more flexibility compared to proprietary options. It highlights that while OSS solutions like Apache Spark, Presto, and others can initially appear cost-effective, the total cost of ownership, including devops and maintenance, often reduces the price advantage over proprietary products like Snowflake. Additionally, the use of vendor-managed OSS can still result in vendor lock-in due to custom distributions and integration with vendor APIs. The text also questions the necessity and feasibility of customizing OSS solutions for most businesses, given the capabilities of serverless offerings. It emphasizes that user experience is often superior in proprietary products, which can be crucial in environments where analytics drive business decisions. Ultimately, the choice of OSS should align with specific business goals and needs.

In the pursuit of an effective analytics infrastructure, this article outlines a comprehensive framework that breaks down the "value chain" of data, from its origin to its integration into decision-making narratives. It emphasizes the importance of carefully selecting tools and processes for each stage of the data lifecycle, including specification, instrumentation, collection, integration, data warehousing, transformation, quality assurance, data discovery, and analysis. The author, drawing from extensive experience with companies like Autodesk, Lyft, and Phantom Auto, recommends tools like Segment for event tracking and integration, while also discussing alternatives such as GCP Pub/Sub and AWS Managed Kafka for cost-effective data collection. The piece underscores the necessity of a scalable and adaptable data infrastructure, highlighting tools like BigQuery and Snowflake for data warehousing and transformation, and stresses the value of making informed decisions about what data to track, rather than merely focusing on technology choices.