Company
Date Published
Author
Henry Franks
Word count
1603
Language
English
Hacker News points
None

Summary

Climate Policy Radar is an NGO dedicated to making global climate laws and policies accessible by transforming vast amounts of government documents into structured, searchable data, serving an annual user base of 350,000, including academics, policymakers, and journalists. They partnered with institutions like the Grantham Institute and UNFCCC to curate a comprehensive dataset for understanding global climate policy. The organization faced orchestration challenges using AWS Step Functions, prompting a shift to Prefect, which provided greater flexibility, scalability, and ease of use for their data scientists and engineers. Prefect enabled the creation of complex workflows, including a document processing pipeline and an AI-powered knowledge graph system, while allowing contributions from various teams without complex setup. This transition empowered data scientists to operate independently, facilitated large-scale research collaborations, and prepared Climate Policy Radar for future challenges like integrating 50,000 climate litigation documents in partnership with Columbia University's Sabin Center. Prefect's implementation streamlined their operations, allowing the organization to focus on core business logic and expand their AI capabilities, demonstrating how an effective orchestration platform can aid in addressing global climate challenges.