June 2022 Summaries

Test data generation is a crucial process for developers and data engineers to create new data that replicates an original dataset. This technique is used for testing applications, developing features, and even training machine learning (ML) models. The importance of test data lies in its ability to evaluate software applications and improve ML/AI algorithms. Techniques for generating test data have evolved over time, from dummy data to the current gold standard: synthetic data. Synthetic data is inexpensive, unlimited, and can be pre-labeled to fit any purpose or use case. It also mitigates edge-case failures by allowing fast, targeted additions to a dataset with each iteration.

The article discusses the use of synthetic time series data and introduces DoppelGANger, a generative adversarial network (GAN) model for generating such data. It highlights the challenges in creating synthetic time series data due to the additional dimension of time and trends across time. The PyTorch implementation of DoppelGANger is presented as an open-source solution that provides flexibility and high-quality synthetic data generation. The sample usage demonstrates how to train and generate synthetic data using both pandas DataFrame and numpy arrays inputs. Results show that the PyTorch implementation produces high-fidelity synthetic data with temporal correlations at different scales, comparable to the original TensorFlow 1 implementation but with a significant runtime speedup (~40x).

Gretel implemented a practical attack on synthetic data models to evaluate their ability to protect sensitive information, using a method described in Carlini et al.'s work on neural networks. The study involved a credit card fraud detection dataset with sensitive features like the last four digits of credit card numbers and cardholder names. The team tested four models with varying privacy settings, including a Vanilla Model, a DP Model, a Noisy DP Model, and a Filtered DP Model, each assessed for their ability to resist memorization of secret values, or "canaries." Results showed that while the Noisy DP Model offered the best privacy protection, it sacrificed accuracy, whereas the Vanilla Model achieved high accuracy but with less privacy protection. The DP Model struck a balance between accuracy and privacy, making it suitable when both aspects are important. The study highlights the trade-offs between privacy guarantees and model performance, suggesting that model selection depends on user priorities.