How Sigma streams query results with Arrow and gRPC

Sigma has enhanced its data processing capabilities by transitioning from JSON to Apache Arrow, a columnar memory format, and adopting gRPC for more efficient service communication. This upgrade has significantly improved the performance and scalability of Sigma's platform, allowing users to interact with their data in cloud data warehouses like Snowflake and BigQuery more efficiently. The changes have resulted in a threefold increase in data movement from the warehouse to the browser, a fivefold reduction in memory usage during data transformations, and a substantial reduction in latency, both from the browser to the warehouse and within the browser itself. Additionally, the use of streaming and chunking further optimizes data handling by reducing memory spikes and processing power requirements. These improvements have not only stabilized the platform by reducing occurrences of out-of-memory crashes but also positioned Sigma to support larger customer bases and organizations more effectively.