March 2023 Summaries

DuckDB` is not a database for ducks, but rather an open-source relational database that allows users to work with data on their laptops and can be easily scaled up to larger storage systems. It provides a query engine that can be used on top of various data storage solutions, including object storage services like Amazon S3. The name `DuckDB` is derived from the idea of creating a "data pond" that can grow into a data lake, with users able to export their data to Parquet files and store them in cloud-based storage services. While some users may joke about ducks using the database, it is not related to the actual use of the software by humans. The most engaged users on the DuckDB community include individuals who have contributed to the open-source project or have shown enthusiasm for its functionality.

The DuckDB project has reached 1 million downloads per month, with users drawn to its ease of use, speed, and cost-effectiveness. Many "normie" users appreciate its Python wrapper, ability to be dropped into any Python script or notebook, and integration with other libraries. Local data workflows are gaining popularity, with users leveraging DuckDB to load data locally, query GCP or S3 buckets from a local machine, and create notebooks or data apps that showcase their work. The community is exploring ways to use DuckDB's columnar query engine for analytics, particularly for aggregation of event data and "one big table" architectures. Users often bring the database engine to their data, using AWS Lambda or Google Cloud Function to process data close to the engine, and some are experimenting with building cost-effective data engineering pipelines. Additionally, some users are dropping DuckDB into their BI tooling, such as Metabase, which supports on-demand embedded DuckDB for always-up file storage.