August 2021 Summaries

This blog discusses using Fluent Bit to collect logs from applications running in an Amazon Elastic Kubernetes Service (EKS) cluster. Fluent Bit is a subproject of Fluentd, developed under the Cloud Native Computing Foundation umbrella, designed for unifying and simplifying cloud logging infrastructure with a smaller memory and CPU footprint compared to its parent project. The author explains how Gretel utilizes FluentBit to collect logs from containerized workloads and publish them to storage and indexing services, making operations simpler by enabling query-based alerts, debugging issues, and monitoring infrastructure trends. The blog then delves into the setup process using Terraform and Helm for installing and versioning FluentBit in a Kubernetes cluster. It covers configuring FluentBit to capture data needed and send it where required, including setting up Inputs (Tail Plugin), Parsers (docker Parser), Filters (Grep Filter and Kubernetes Filter), and Outputs (Amazon Cloudwatch service). The author also highlights the importance of ensuring Node permissions for network access and log operations in Amazon's IAM configuration. Finally, the blog provides a GitHub repository link where readers can see FluentBit set up on an AWS EKS cluster to observe its functionality in action.

In this blog post, a synthetic data pipeline is built using Apache Airflow, Gretel's Synthetic Data APIs, and PostgreSQL. The purpose of the pipeline is to extract user activity features from a database, generate a synthetic version of the dataset, and save it to S3 for use by data scientists without compromising customer privacy. The pipeline consists of three stages: Extract, Synthesize, and Load. Gretel's Python SDKs are used to integrate with Airflow tasks, and an example booking pipeline is provided along with instructions on how to run it end-to-end.

Using Python, Pandas, and Gretel-synthetics, a process is outlined to create an anonymized version of a dataset by generating synthetic data from a CSV or Pandas DataFrame. The tutorial involves setting up an environment using the Gretel API, accessing a template configuration to define neural network parameters, and loading sample data into a DataFrame. The model is then trained using Gretel's cloud service, which allows for efficient processing without local GPU requirements. The synthetic data generated is evaluated for its ability to mimic the original dataset's characteristics, correlations, and distributions, achieving a high accuracy score with no duplication of the original data. Techniques such as PCA and field-by-field comparisons are employed to assess the synthetic data's fidelity in replicating the statistical properties of the input data, ensuring privacy and providing flexibility for generating additional records.