Understanding Iceberg Support in the Databricks Ecosystem

Spark has played a significant role in the development and excitement around Iceberg, leading to strong support for Iceberg in the Databricks ecosystem, which is primarily built on Spark. Databricks' Unity Catalog has enhanced governance capabilities and now natively supports Iceberg as part of its catalog, allowing integration with the UniForm feature to generate Iceberg metadata tables alongside Delta Lake tables without conversion. However, UniForm compatibility has limitations, such as issues with deletion vectors and lack of support for Iceberg Views. Databricks recently introduced OAuth support for the Iceberg Catalog REST API, enabling direct access to Iceberg tables, though writing capabilities remain unsupported. Challenges exist with querying nested namespaces due to Databricks' handling of the unit separator, indicating the early but rapidly evolving state of the technology. In June 2024, Databricks acquired Tabular, an Iceberg data catalog, signaling potential future synergies in their offerings, though Tabular is no longer accepting new signups.