March 2024 Summaries

In this blog post, the author discusses how to integrate and validate NYC real estate sales data from multiple sources, including GlareDB Cloud, local files, and Postgres, with NYC tree census data stored in Snowflake to explore the correlation between sold properties and nearby trees. The process involves using Great Expectations (GX) to perform data quality checks and create an Expectation Suite, which ensures the data meets certain criteria before being loaded into tables. The author provides a step-by-step guide on setting up a GlareDB connection, joining data from different sources, and using GX to validate data assumptions, such as the number of trees near properties. The post emphasizes the importance of these validations by demonstrating how assumptions about the data can be tested and adjusted when new sales data becomes available, thereby ensuring data integrity before integration into larger data pipelines. Additionally, the author hints at future posts exploring further integrations with data tools like dbt and invites readers to engage with GlareDB through various platforms.

GlareDB's v0.8.0 introduces "workload sharding" features to facilitate the efficient processing of data by multiple stateless application servers, ensuring equal data distribution without redundancy. The method involves hashing a field with high cardinality, such as a user ID, into a 64-bit integer and using modulo operations to distribute workloads evenly among workers, minimizing the need for coordination and infrastructure. However, this approach presents challenges, including difficulties in adjusting the worker pool size and potential issues with workload reallocation if a worker fails. To support this sharding method, GlareDB has added fnv() and siphash() functions, which generate uint64 values, and a partition_results function, allowing users to easily implement this strategy in queries. This function works by returning a boolean that can be used in a WHERE clause, facilitating the assignment of tasks to specific workers based on partition IDs.

The blog post details the process of creating an interactive dashboard using Streamlit to visualize NYC real estate sales data stored in GlareDB. It guides readers through setting up a Streamlit application, connecting to GlareDB, and utilizing Python and Pandas to manipulate and display the data. The tutorial demonstrates how to generate a line chart to show the trend of sales over time and a map to illustrate sales distribution by zip code, using pgeocode to convert zip codes into latitude and longitude coordinates. The post also covers enhancing the dashboard's aesthetics by organizing visualizations into columns, adjusting layout settings, and adding headers for clarity. The resulting dashboard offers users a visual representation of real estate sales trends over time and geographical distribution across NYC, with suggestions for further enhancements like interactivity and data quality checks.