May 2025 Summaries

Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi provide a layer of abstraction over raw data files, enabling features such as ACID transactions, schema evolution, and time travel. These specifications define how to store and manage large datasets in a structured manner on distributed storage systems, providing consistency and reliability across multiple processing engines. The primary open table formats in use today are designed to solve distinct challenges, with Iceberg shining in analytics scenarios, Delta Lake best suited for Spark environments, and Hudi optimized for streaming-centric environments. Iceberg addresses the need for database-like capabilities in object storage systems, enabling reliable access to data that would otherwise be locked in raw files. Its key features include ACID transactions, schema evolution, time travel, and interoperability, making it an essential component for modern data platforms that need to scale. The combination of Iceberg and Amazon S3 offers a potent alternative to traditional proprietary data lakes and data warehouses, enabling companies to offload their data to lower-cost object storage systems while still being able to query and use the data as if it were in a more traditional data environment.

Postgres has become a standard due to its great product, open source dynamics, and the inability of any single entity to own it. The three emerging standards in the data world are Postgres for OLTP databases, Open Telemetry for telemetry, and Iceberg for OLAP/warehousing. These standards exhibit the same open source tenets as Postgres, including an OSI-approved license, self-hosting capabilities, and commercialization by multiple entities. The industry is shifting left, with developers taking on more responsibility for observability and warehousing, driving the adoption of these standards. S3 is emerging as the ultimate data infrastructure, enabling interoperability between various tools and driving the concept of "S3 as a database". Supabase is focusing on integrating Postgres, OTel, and Iceberg into its platform to cater to the growing demand for open data standards.

Supabase offers a different path to building modern applications by exposing a secure, auto-generated REST and GraphQL API for every table, view, and stored procedure in the public schema of your Postgres database. This reduces weeks of infrastructure work into minutes without sacrificing flexibility or control. Supabase's API layer compresses complexity, allowing developers to focus on product development while maintaining control over their data. The platform provides auto-generated client libraries, supports features like filters, pagination, and GraphQL, and offers tools for building custom API endpoints through Database Functions and Edge Functions. By eliminating the need for manual middleware, Supabase simplifies architecture, reduces technical debt, and cuts cloud spend, providing a unified data plane for application logic, internal services, and external integrations. The platform also ensures security through Row Level Security (RLS) policies enforced by the database itself, authentication via JWTs, and regional project hosting with Data Processing Agreements and more.

Event triggers in Postgres allow for powerful database automation, but only superusers can create them. To make event triggers accessible to regular users in a safe way, the `supautils` extension uses a combination of the Utility Hook and Function Manager Hook. A "privileged role" is used as a proxy to superuser capabilities, allowing regular users to create event triggers while preventing privilege escalation. The privileged role intercepts event trigger creation, elevates it to a superuser, creates the trigger, and then downgrades back to the privileged role, ensuring safety. However, this approach still has limitations, such as targeting every role and running with target role privileges. To address this, the extension also skips user event triggers for superusers and "reserved roles" used in managed services like `pgbouncer`. User event triggers are now available through the `supautils` extension, allowing regular users to create event triggers safely without superuser access, and the feature is being discussed for inclusion in Postgres core.