February 2025 Summaries

Bodo is an open-source, high-performance compute engine that simplifies scaling Python workloads from laptop to cluster without major code changes. It uses innovative auto-parallelizing just-in-time (JIT) compiler technology and relies on MPI-based high-performance computing (HPC) technology to make it both easier to use and often orders of magnitude faster than tools like Spark or Dask. PyIceberg is an open-source Python library that provides helpful functions to manipulate Iceberg databases, including creating and deleting Iceberg tables, reading and writing to Iceberg tables with data libraries and compute engines such as Pandas, DuckDB, Polars, Ray, and so on. By integrating PyIceberg into Bodo, developers can now leverage Apache Iceberg without Java overhead, making it easier to scale from local development to full-cluster execution without switching tools or rewriting code.

The text discusses the challenges of processing large-scale genomic data in bioinformatics, particularly when dealing with multiple large FASTA files. To address this issue, a high-performance Python framework called Bodo is introduced, which provides a way to parallelize computationally intensive tasks efficiently. The example demonstrates how to use Bodo to read multiple FASTA files in parallel and extract sequence IDs alongside their peptide strings, producing a single CSV file containing all the annotation, probe, and source information derived from each input. This approach can significantly expedite data processing, especially on larger datasets, allowing bioinformaticians to focus more time on deriving meaningful insights from their sequence data.

AWS S3 Tables is a new service that simplifies the usage of Iceberg open table format by providing automatic setup and maintenance, including built-in table maintenance and optimization. This service mainly relies on Apache Spark, which requires complex configuration and a Java-based engine. Bodo, an open-source high-performance compute engine for Python data processing, addresses this challenge by offering simple S3 Tables usage in both Python and SQL, simplifying the read and write of large tables with high performance and efficiency. Bodo enables efficient querying and processing of S3 Tables without the need for complex infrastructure or data movement, allowing users to perform analytical queries, aggregations, and transformations on S3 Tables using familiar Pandas APIs. To use S3 Tables, users need active AWS credentials and the necessary dependencies installed and upgraded, as well as the AmazonS3TablesFullAccess policy attached to their credentials. Bodo provides a simple way to write dataframes with its `to_sql` method and read tables into dataframes using the Pandas `read_sql_table` function. This feature is available in Bodo pip and Conda releases starting from 2025.1, and users can try it out by installing Bodo with pip or Conda.