January 2025 Summaries

Bodo is an open-source, high-performance compute engine for Python data processing that simplifies scaling Python workloads without major code changes. It uses an innovative auto-parallelizing just-in-time (JIT) compiler to deliver very high parallel performance to the whole workflow. Bodo's new `@bodo.wrap_python` decorator makes it easy to run libraries in parallel while using Bodo, allowing users to keep using their favorite Python libraries in parallel without extra code changes. This feature is similar to Numba's objmode but has a simpler function interface and can be significantly faster. It is available in Bodo pip and Conda releases starting from 2024.12.3 and can provide a 5x speedup on local M1 Mac with a sample dataset, scaling to any cluster and data size as well.

Bodo has made its compute engine open source under the Apache license, bringing high-performance computing capabilities to all data and AI workloads. By parallelizing Python workloads, Bodo delivers extraordinary performance, accelerates AI pipelines, and significantly reduces cloud costs. The company's mission is to make HPC as simple as possible, and by sharing its power with the community, it aims to empower everyone to scale their Python workloads effortlessly. With a focus on openness and shared progress, Bodo is committed to being an integral part of the large open-source ecosystem surrounding Python, and its journey has been influenced by projects like Numba. The Bodo Compute Engine transforms standard Python code into efficient, parallelized execution without requiring changes to the codebase, making it possible for data engineers, data scientists, and AI/ML practitioners to accelerate data processing, model training, optimize analytics pipelines, and reduce cloud and energy costs.

Python has become a popular choice for data engineers and data scientists, but scaling Python code efficiently remains a challenge. Compute engines like Bodo, Spark, Dask, and Ray/Modin aim to bridge this gap, offering Python scaling while striving for high performance. A recent benchmark tested the performance of these engines on a Python program that computes the summary of monthly trips with precipitation data on the NYC Taxi public dataset. The results reveal massive performance differences: Bodo delivered a 20x speedup over Spark (95% cost savings), 50x over Dask (98% cost savings), and a staggering 250x over Ray/Modin (99% cost savings). This is attributed to Bodo's HPC-based compiler approach, which differs from the distributed task scheduling design of other engines. The benchmark was conducted on a 4-node cluster setup on AWS, using a smaller subset of the dataset to allow local execution on a laptop. Bodo shows a roughly 4x improvement over Pandas, while other engines can be substantially slower than regular Pandas. The advantages of Bodo's architecture and design make it a strong competitor to existing engines like Spark, Dask, and Ray, offering unparalleled speed, ease-of-use, and cost efficiency for compute-heavy workloads.