March 2025 Summaries

Apache Spark has long been a critical tool for large-scale analytics and machine learning, but it has faced challenges in achieving comprehensive data lineage due to its unique nature. Existing solutions like Databricks Unity Catalog and OpenLineage only offer runtime lineage, complicating impact analysis for code changes. Foundational has introduced an advancement with a new capability that automates data lineage extraction directly from Spark code, including PySpark, Scala Spark, and Spark SQL. This code-based approach not only provides more detailed and accurate lineage information but also offers proactive insights by analyzing pending code changes and pull requests to identify potential data issues before deployment. Foundational aims to offer end-to-end visibility by connecting Spark pipelines with upstream sources and downstream BI tools, thus enhancing transparency, governance, and confidence in managing Spark-driven data initiatives.

Foundational, a powerful data governance platform, is now available on the AWS Marketplace, making it more accessible for AWS users to purchase and implement. This development simplifies the procurement process and provides data teams with faster access to tools essential for ensuring data trust, accuracy, and governance. Foundational offers comprehensive transparency, traceability, and accountability across the data lifecycle, which is crucial for data leaders focused on maintaining data trust and complying with regulatory requirements. By using Foundational through AWS, organizations can accelerate their data governance initiatives, improve data quality, enhance decision-making, and ensure stronger compliance. Key benefits include end-to-end data visibility, improved data quality and trust, streamlined data governance, and increased operational efficiency. Data teams can track data flows, ensure data accuracy, visualize data dependencies, and perform impact analyses to prevent workflow disruptions, thereby unlocking the full potential of their data.