October 2021 Summaries

The text discusses Kafka Streams, an abstraction over Apache Kafka producers and consumers that allows developers to focus on processing data without worrying about low-level details. It explains the concept of processor topology, where event stream data flows through nodes with custom-logic operations, and how state stores enable tracking latest values by key. The passage also covers windowing in Kafka Streams, which gives users a snapshot of an aggregation over a given time frame, allowing for time-based analytics. Additionally, it touches on the Processor API, which provides more flexibility than the standard DSL but requires specifying topology node by node, and scheduling arbitrary operations using punctuations. The text concludes by discussing error handling mechanisms, parallelism, and state store persistence, highlighting Kafka Streams' ability to achieve scalability through dynamic workload redistribution among instances.

The Oracle CDC Source Premium Connector v1.3.0 now supports Oracle 19c, allowing customers to leverage the connector's capabilities in their production environments. The connector uses LogMiner to read redo logs from Oracle Database and has overcome challenges related to handling DDL statements and schema changes. Users can configure the connector by setting the "oracle.dictionary.mode" parameter and follow specific steps for addressing DDL changes. A CDC pipeline can be created between Oracle Database and Azure Synapse Analytics using the self-managed Oracle CDC Source Connector and fully managed Azure Synapse Analytics Sink Connector, enabling customers to capture raw Oracle events and write them to table-specific topics in Confluent Cloud. The connector also supports various destinations like Snowflake, MongoDB Atlas, Google BigQuery, and Amazon Redshift.

Confluent has introduced three new features for its fully managed connectors: single message transforms (SMTs), connect log events, and connector data output preview. These enhancements enable users to transform, monitor, and preview connector data, improving operational transparency and providing additional insights during the data integration journey. Single message transforms allow lightweight modifications to message values, keys, and headers, while connect log events expose connect logs for visibility and troubleshooting. Connector data output preview provides a dry-run functionality for connectors, allowing users to see an accurate representation of record structure before actual ingestion. These features aim to provide a better user experience and enable more efficient use of Confluent's fully managed connectors.

The text discusses how Confluent's data warehouse modernization solution helps enterprises connect their on-premise and cloud-based data warehouses, enabling real-time ETL to reduce costs and improve time to value. It mentions that Confluent allows preprocessing of data in stream with ksqlDB through simple SQL statements and offers a rich ecosystem of 120+ connectors for rapid integration. The solution works with any cloud data warehouse and can help migrate data from on-premise data stores into the new cloud-based data warehouse. Confluent's Cluster Linking technology enables easy bridging between environments, reducing complexity and improving ROI of analytics data.

Confluent and Red Hat have collaborated, resulting in a complete cloud-native experience for application developers and platform operators to set data in motion on Red Hat OpenShift. Confluent for Kubernetes is a certified Red Hat OpenShift operator available in the Red Hat Ecosystem Catalog, offering benefits such as streamlined operations, built-in security features, and consistent deployment across various environments. The partnership also includes Build with Confluent, which supports system integrators in developing joint solutions faster, and access to Confluent's Go-To-Market teams for amplifying their offerings in the market.