Company
Date Published
Author
Lucia Cerchie, John Neal, Josep Prat
Word count
1679
Language
English
Hacker News points
None

Summary

The "data explosion" poses significant challenges, including finding a suitable place for data storage and retrieval. To address this, Apache Kafka and Confluent can be used to keep data in motion longer, creating pipelined architectures that utilize data generated upstream by downstream consumers. Qlik Replicate is an enterprise-class solution that retrieves changes from relational databases in real-time, supporting high-volume production loads and delivering data in Avro or JSON formats to Kafka targets. By configuring a Qlik Replicate Kafka target, users can customize message payload, omit data columns, and use the Confluent Schema Registry. Feedback loops are also essential in streaming architectures, particularly those involving machine learning steps, where upstream targets send information to components further upstream, affecting subsequent processing.