The Semantic Layer Problem Nobody Wants to Talk About

The discussion between Michael Driscoll and Lloyd Tabb highlights the challenges and misconceptions surrounding the development of semantic layers in data analytics. Tabb, who has extensive experience from his work with Looker and now Malloy, argues that many semantic layer tools treat data practitioners as less skilled than developers, using simplified interfaces that fail to address the complexities of data transformation. He emphasizes the importance of respecting the craft of data development and providing robust tools akin to those used by software engineers. Tabb identifies a fundamental flaw in conventional semantic layers that flatten data into simple tables, which can lead to inaccurate results when complex joins are required. Instead, Malloy preserves the graph structure of data, allowing for more accurate and flexible queries. He also notes a shift in infrastructure needs, where powerful local query engines like DuckDB can handle many workloads previously requiring cloud-based solutions. While there is significant interest in integrating AI with semantic layers, Tabb is skeptical of approaches that forego semantic modeling, as they often lead to failure due to the chaotic nature of enterprise data warehouses. Despite the entrenched use of SQL, Tabb believes a new language like Malloy, which abstracts the complexities of SQL dialects, could eventually supplant it, provided it is learnable and productive for developers. Ultimately, the success of a semantic layer depends on rigorous development practices, comprehensive tooling, and a deep understanding of the data development process.