June 2021 Summaries

The panel discussion at Datanova, moderated by Andy Cotgreave, explored the evolving landscape of data analysis, emphasizing the declining reliance on dashboards and the growing importance of exploratory data analysis and predictive analytics. Panelists including Charles Wilson, Parker Dillion, and Manav Gupta highlighted the shift towards using data sets to address new questions and the need for data analysts to acquire technical skills and integrate into business units as proactive thought partners. While artificial intelligence is acknowledged as a potentially transformative tool, its practical application remains challenging, though it can enhance the curiosity-driven aspect of data analysis. The discussion also delved into whether data analysis is an art or a science, concluding that it is a blend of both, requiring creativity and scientific rigor. As the role of data analysts becomes increasingly crucial, they must embrace critical thinking and problem-solving to drive business success.

Comcast has developed an advanced hybrid analytics platform that efficiently integrates diverse data storage systems, enabling seamless data access for users across both on-prem and cloud environments. Initially faced with data silos spread across traditional RDBMS, data warehouses, NoSQL databases, and data lakes, Comcast's engineering team, led by Bryan Aller and Pavan Madhineni, sought a scalable solution to unify data access without relying on extensive ETL processes. By implementing a query engine using Trino and later integrating Starburst's enterprise-ready connectors, they created a Trino-to-Trino connector allowing for seamless communication between cloud and on-prem instances, effectively managing identity and security while abstracting data storage complexities. This innovation has enabled Comcast to leverage a data mesh approach, allowing analysts and data scientists to perform real-time analytics with standard SQL without duplicating data. The platform supports high-performance queries, significantly reducing processing times, and has been instrumental in handling large-scale data pulls of 250-300 TB daily, demonstrating both scalability and speed for hundreds of users accessing thousands of datasets.

The integration of Starburst and Immuta facilitates rapid and controlled access to data while maintaining privacy and compliance with regulations like CCPA, GDPR, and HIPAA. This collaboration enables data managers and engineers to address the dual challenge of providing swift data access and ensuring the protection of sensitive information. Immuta enhances the capabilities of Starburst by offering Attribute-Based Access Control (ABAC), which refines traditional Role-Based Access Control (RBAC) with more granular specifications based on user attributes, and Purpose-Based Access Controls that tie data access to specific business purposes. These features allow organizations to implement detailed access policies that align with corporate and regulatory requirements without delaying data delivery. Immuta also simplifies the process of tagging sensitive data fields through automatic detection, making it easier to apply protective measures. By routing all data access through Immuta, Starburst ensures that users can efficiently query federated data across multiple sources while adhering to customized privacy and access policies tailored to their business roles.

In the "Trino on Ice IV: Deep Dive Into Iceberg Internals" blog post, the implementation details of the Iceberg table format in conjunction with the Trino query engine are explored, building on earlier posts in the series. The article delves into the structure and function of various files, such as metadata and snapshot files, generated during Iceberg table operations, using tools like Trino, Avro tools, and MinIO. It emphasizes the importance of understanding these files for troubleshooting and highlights the process of inspecting snapshot files with Avro tools to better grasp how Iceberg manages data changes. Additionally, the blog discusses the use of the Hive metastore with Iceberg in Trino, illustrating the convenience of migrating from Hive to Iceberg tables due to existing support. The post underscores the significance of manifest lists and their role in managing Iceberg's persistent tree structure, which supports features like concurrency and time travel across data snapshots.

Managing secrets in Trino is crucial for maintaining security and preventing sensitive data exposure, particularly as configuration files, often stored in source control systems, can unintentionally reveal information such as usernames and passwords. To mitigate risks, Trino supports using environment variables as configuration property values, allowing sensitive information to be managed securely without being stored in plaintext. When installing Trino, environment variables can be defined in a CONFIG_ENV file, typically set by a startup script, ensuring secure configuration during service start-up. External secret management tools like Google’s Secret Manager, AWS Secrets Manager, and Hashicorp Vault can be integrated to retrieve secrets securely. For instance, secrets stored in AWS Secrets Manager can be accessed via the AWS Console or CLI and included in Trino’s configuration through environment variables set in the /etc/trino/env.sh file. Additionally, Kubernetes can use secrets as environment variables, and Starburst’s helm charts support referencing external secrets, thus enhancing security and reducing the risk of data breaches.