August 2023 Summaries

CarTrawler, a car rental API provider, has significantly enhanced its data operations by integrating Soda into its technology stack, which includes platforms like Snowflake and AWS, to ensure scalable and self-service data quality management. Founded in 2004 in Dublin, CarTrawler supports car hire functionality for major airlines and travel agencies, managing vast amounts of data from numerous suppliers and partners. Faced with challenges in data complexity and scalability, CarTrawler initially attempted to build its own data quality tool but found it lacking in self-service capabilities and automation. By adopting Soda, CarTrawler empowered its data engineers to test data quality as code and allowed business users to manage their data quality expectations independently. This transformation has led to more efficient data operations, reduced reliance on engineering for quality checks, and improved data reliability, contributing to a 5% increase in revenue per visitor. The integration of Soda has not only created a single source of truth for data analytics but also democratized data management by allowing users to apply data quality rules themselves, fostering trust and accountability across the organization.

In the realm of data management, ensuring high data quality is critical, and Soda's innovative check suggestions offer a streamlined approach to achieving this. By automating basic data quality checks with its Soda Checks Language (SodaCL), Soda enables users to systematically validate data integrity without the burden of starting from scratch. The "soda suggest" feature allows users to generate production-ready data quality checks by profiling datasets and recommending relevant validations for completeness, freshness, and format validity, among others. Users can customize these checks to fit their specific needs, thus preventing data issues from impacting downstream processes. Additionally, Soda is enhancing its offerings to include more detailed checks and user-friendly interfaces, aiming to accommodate both technical and business-oriented users. This initiative not only simplifies the implementation of data quality coverage but also encourages the adoption of best practices across data teams, ultimately minimizing disruptions caused by poor data quality.

Soda's approach to data quality emphasizes the importance of performance optimization to prevent cost escalation and maintain trust in data systems. By providing full configurability through YAML configuration files, engineers can manage data quality checks with precision, allowing for efficient resource use and cost control. Soda recommends executing checks only on relevant data slices, thereby reducing unnecessary data processing and associated costs. The platform also encourages grouping multiple checks into single queries to minimize passes over data, further optimizing costs. Additionally, leveraging compute engine-specific features like query caches ensures faster and more cost-effective data profiling. This configuration-first strategy empowers engineers to balance data quality coverage with cost efficiency, ultimately helping to control data warehouse expenses while scaling quality checks across teams.