December 2015 Summaries

Kafka Connect is designed to make it easier to build large-scale, real-time data pipelines by standardizing how you move data into and out of Kafka. It uses connectors to read from or write to external systems, manage data flow, and scale the system without writing new code. Kafka Connect manages common problems in connecting with other systems, such as scalability, fault tolerance, configuration, and management. The JDBC connector allows importing data from any relational database with a JDBC driver into Kafka, while the HDFS connector exports data from Kafka topics to HDFS files in various formats and integrates with Hive for immediate querying with HiveQL. Kafka Connect can perform database change capture, schema migration, and custom partitioning, making it easier to build scalable ETL pipelines.

The Confluent Platform 2.0 is now available, offering a centralized stream data platform for organizations that want to build streaming data pipelines around Apache Kafka. It includes new features such as multi-tenant support, data flow from existing systems to Kafka, and improved security capabilities including client authentication, user-defined quotas, and encryption of network connections. The release also introduces Kafka Connect, a tool for large-scale real-time data import and export between Kafka and various other systems. Additionally, Confluent Platform 2.0 includes open-source connectors for log files, relational databases, and Hadoop, as well as fully-featured native Java producer and consumer clients with feature and API parity to Apache Kafka 0.9 Java clients. The platform is entirely open source and backed by subscription support, expert training, and technical consulting services.