May 2022 Summaries

BodoSQL is a SQL engine that bridges the gap between Python and SQL, providing type checking, error checking, end-to-end optimization, and parallelization across both languages. It addresses the "two-language" problem in data applications, where developers often use native Python/Pandas for large-scale data processing without the need for SQL, but legacy code may still rely on SQL. BodoSQL is designed to eliminate the need for code rewrites or additional developer training, and it offers extreme performance and scalability, making it suitable for large-scale datasets and clusters. The engine uses a high-performance computing parallel architecture with MPI for execution, delivering better performance than other SQL engines like SparkSQL. It also provides cross-language optimization capabilities, eliminating unnecessary columns from computations and avoiding reading them from storage altogether. Additionally, BodoSQL offers end-to-end error checking at compilation time, improving robustness and reducing the risk of production jobs failing due to runtime errors.

The text discusses using a platform called Bodo for machine learning practitioners to rapidly explore data and build complex pipelines. Bodo allows developers to scale their codes from laptops to its platform, making it easy to distribute data across processes. The example uses the 2019 NYC Yellow Cab trip record dataset, exploring the data, removing outliers, and computing descriptive statistics before building a regression model to predict taxi tips using features such as tolls_amount, trip_distance, and fare_amount. The code uses Bodo's parallel execution model, which distributes the data across processes for efficient computation. After training the model, it evaluates the performance with RMSE and R^2 scores, providing a first estimate of its quality. The text also mentions that Bodo provides an extensive API and supports many well-known packages to accommodate various machine learning pipelines.

Bodo is a platform that enables machine learning practitioners to rapidly explore data and build complex pipelines. It allows developers to scale their codes from laptops to the Bodo platform, making it easy to distribute data across processes for parallel execution. The platform is used in this example to detect potential cases of credit card fraud using classification techniques from machine learning. The dataset consists of transactions made by European cardholders in September 2013 and has a strong imbalance between fraudulent and non-fraudulent records. To mitigate this, the dataset is sub-sampled to achieve a balanced distribution. Features are analyzed to identify those that correlate meaningfully with the output, and weakly-correlated columns are dropped. A Random Forest Classifier is trained on the rebalanced data, and its performance is evaluated using metrics such as the Receiver Operating Characteristic (ROC) curve and confusion matrices. The model's performance appears to be good, but oversampling the dataset can further improve it. Bodo provides a platform for building and deploying machine learning models, making it easier to scale and distribute code across processes.

Bodo aims to make high-performance computing accessible by integrating Python/pandas with SQL, leveraging existing open-source innovations from communities like Numba, MPI, and pandas. The platform uses a JIT compiler to achieve C++/MPI levels of speed and scalability. Bodo's approach is community-driven, utilizing open-source projects while creating commercially available optimized software and enterprise tools. By standing on the shoulders of giants, Bodo collaborates with existing open-source communities, fostering a collaborative environment where developers from different backgrounds work together to address mutual problems. The platform prioritizes transparency, publicly available source code, and multi-organizational engagement, promoting an "open" and interoperable ecosystem that supports standard protocols, file formats, storage systems, workflow orchestration tools, and more. By leveraging open-source software, Bodo aims to increase compatibility, facilitate data exchange between platforms, and provide a seamless user experience. The company believes that community-driven open source is critical for effective modern data processing platforms, as no single company can solve all data problems alone.

Snowflake Data Cloud simplifies data management for data engineers at a near-unlimited scale, while the Bodo Platform brings extreme performance and scalability to large-scale Python data processing. Snowflake and Bodo have combined forces to enable data teams to complete very large ETL and data-prep jobs in a fraction of the time with better efficiency than they could with Snowflake alone. This is achieved through Bodo's supercomputing-style MPI-based parallel approach to data analytics using native Python, which is an order-of-magnitude faster and far more resource-efficient than using Apache Spark. Combining these best-in-class storage and compute solutions requires very efficient data movement between the two platforms, which has been a key focus of Bodo's partnership with the Snowflake team. Bodo's Snowflake Ready connector lets Bodo clusters read terabytes of Snowflake data extremely fast, built-in and fully automatic while delivering high performance similar to Parquet on S3 datasets. Reading large amounts of data from Snowflake using Bodo is demonstrated to be faster than reading the same data from Parquet files from AWS S3, with Bodo taking under 3 minutes to read 1TB of data compared to Snowflake taking around 154 seconds. The Bodo JIT compiler optimizes queries by selecting only necessary columns and pushing down filters automatically, which simplifies data pipelines. Additionally, Bodo's distributed fetch mechanism reads data into parallel chunks on which cores will execute the application in parallel, enabling developers to build highly performant data pipelines with fast distributed fetch and parallelized computation.