Company
Date Published
Author
Sam Kleinman
Word count
1393
Language
English
Hacker News points
None

Summary

GlareDB, in its version 0.8 update, has introduced support for BSON files, enhancing its capabilities to read and write data both locally and to cloud storage. BSON, known for its use in MongoDB, is a binary form of JSON with a more complex type system, making it particularly useful for databases due to its row-focused structure and efficient document parsing through length-prefixing. Despite GlareDB's column-focused architecture, BSON's internal allowances align well with GlareDB's support for diverse data formats. Users can utilize BSON in GlareDB as seamlessly as other formats, with functionalities like reading from local paths or cloud services and writing data using the COPY TO operation, which can handle transformations and export data to various storage platforms. Implementation of BSON in GlareDB leverages StreamingTables in DataFusion, optimizing memory usage and latency by streaming data directly through the query engine. While schema inference currently relies on sampling the initial documents, future updates aim to improve schema control and projection capabilities. The integration of BSON not only enhances GlareDB's versatility but also improves its MongoDB connectivity, offering extensive opportunities for users to build innovative solutions.