Using Managed Iceberg Tables to Build a Datalake Inside Snowflake

Snowflake's introduction of Managed Iceberg Tables allows users to construct an Iceberg data lake within Snowflake, utilizing a new object called an EXTERNAL VOLUME for interacting with Iceberg tables in object storage. This feature necessitates setting up IAM controls for Snowflake's access and includes the creation of Iceberg tables within these external volumes, though it does not support direct conversion of existing tables. Users may need to adjust API data types when migrating tables, such as changing TIMESTAMP formats. Snowflake-managed Iceberg tables come with a proprietary catalog that is limited to JDBC connections, restricting external services to read-only access. The catalog includes a CATALOG_SYNC feature that synchronizes changes made in Snowflake with an external catalog in Polaris, which supports Iceberg REST and potentially allows connections between multiple Snowflakes. Despite being relatively new, this managed catalog might eventually be integrated into Polaris, which could expand its utility for Iceberg architectures.