Redacting data with Redpanda Data Transforms

Redpanda 23.3 introduces the capability to perform stateless data transformations within its broker using WebAssembly (Wasm), enabling tasks such as format transcoding, data validation, filtering, and redaction of sensitive information directly within Redpanda. A new feature, Data Transforms, allows users to embed data processing without relying on external platforms, and it includes a pre-built library for config-driven data redaction, which is essential for handling sensitive data in compliance with regulations like GDPR and HIPAA. The process of redaction is simplified with a framework that offers a variety of built-in actions for modifying sensitive data fields, and the introduction of a no-code solution makes it accessible to non-developers. Users can deploy redaction transforms by configuring redaction rules, creating output topics, and running deployment scripts. The redaction framework is extensible for custom use cases, and a demonstration using Docker Compose shows the practical application of these transformations, highlighting how Redpanda facilitates secure and efficient data management.