March 2021 Summaries

AWS recently introduced support for S3 Object Lambda, allowing users to anonymize data at access time using tools such as Gretel Transforms. This blog demonstrates how to launch a real-time S3 object transformation service using the following stack: AWS Lambda with containers, managed by Serverless; AWS API Gateway; S3; and Gretel Transforms. The solution design involves creating an API endpoint for generating pre-signed URLs to fetch and transform S3 objects. To test this solution, a pipeline is constructed using Gretel Transforms, which can be used to transform entire DataFrames. A Docker container is built on the AWS Lambda Python image, containing the handler.py module. The serverless.yml file manages the AWS infrastructure and resources needed for the system.

Gretel has introduced a new synthetic data report, featuring an overall Synthetic Data Quality Score to help assess the quality of synthetic data compared to original source data. The score is based on metrics such as Field Distribution Stability, Field Correlation Stability, and Deep Structure Stability. By analyzing these metrics, users can determine whether their synthetic data maintains the same statistical properties as the original dataset and if it's suitable for specific use cases.

Synthetic data has been tested for its effectiveness in machine learning models against original datasets. Experiments were conducted using popular ML algorithms on several Kaggle datasets and the results showed that synthetic data performed quite well, with similar accuracy to the original data. The use of synthetic data can be beneficial in situations where access to sensitive data is restricted or when there is a lack of relevant data for training models. Synthetic data can also help broaden testing scenarios and improve AI systems' performance.

Gretel, a privacy engineering company founded by engineers, aims to enable developers, researchers, and scientists to quickly create safe versions of data for various purposes such as machine learning workloads, pre-production environments, and data sharing across teams and organizations. The company launched a free public beta six months ago and learned that 50% of its users are software and data engineers, while 40% are data scientists and machine learning researchers. The majority of the users work in technology focused industries, followed by healthcare and finance, education, retail, and government. Gretel's current beta focuses on using a set of SDKs and Python notebooks to execute workloads, but it plans to release an updated version called Beta2 with end-to-end functionality for privacy engineering as a service, making it accessible to non-developers. The company will also focus on being purely API driven, providing REST APIs hosted in Gretel Cloud and supporting various languages and frameworks. Additionally, Gretel plans to double down on open source by bundling all its open source functionality together into an Open Source Python SDK called Gretel Core.