August 2020 Summaries

Convoy has successfully implemented a near real-time ELT pipeline using Apache Kafka® and Snowflake, which has significantly improved data accessibility and timeliness within the company. The new system ensures low latency, zero data loss, and low maintenance, enabling teams to make data-driven decisions more effectively. The design of this system includes Debezium Source Connector for change data capture, Apache Kafka for event streaming, Snowflake Sink Connector for data ingestion, and Snowflake Streams & Tasks for continuous data transformation. This implementation has led to a significant reduction in data warehouse latency, allowing the company to maintain trust in its data sources while scaling operations.

Load testing is crucial for optimizing real-time data streams using Apache Kafka®. It helps determine the system's maximum capacity, breaking points, and recovery capabilities without manual intervention. There are three main types of load testing: stress, spike or surge, and soak tests. These tests help engineers understand their project's limitations before deployment to prevent potential disasters like Robinhood's outage on a historic trading day.

Apache Kafka 2.6 is now available as a managed service on the Aiven platform, offering enhancements across its core features, Kafka Connect, and Streams to make it easier to operate at scale. Key updates include explicit rebalance triggering on the consumer, error reporting in Sink connectors, filter and conditional Single Message Transformations (SMTs), topic-specific settings for new topics, improved producer scalability with exactly once semantics, and smooth scale-out capabilities for Streams applications.

Spare, a global transportation systems provider, has chosen Aiven for PostgreSQL as their application database solution. The company offers a platform for municipalities worldwide to build public transit and micromobility services for citizens. A case study reveals that Aiven for PostgreSQL significantly improved the reliability of Spare's platform while saving them the cost of two full-time engineers. To learn more about other companies using Aiven, visit their case studies page.

The text discusses the importance of careful planning and execution when building an enterprise Internet of Things (IoT) solution. It highlights three fundamental considerations to ensure success: 1) Business vision, which involves defining the purpose of the IoT solution and setting clear objectives; 2) Time-to-market and know-how, emphasizing the need for expertise in both hardware and software development; and 3) Technical and operational setup, focusing on the integration of hardware and data infrastructure. The text also suggests working with quality partners to reduce complexity and cost in IoT development.

Real-time applications are becoming increasingly important in our fast-paced world, underpinning various aspects of daily life from consumer apps to business collaboration tools. To build successful real-time applications, engineering teams should consider a microservice architecture, utilize an open source data streaming platform like Apache Kafka, use self-healing nodes, ensure observability, and incorporate built-in data governance practices. Lenses.io and Aiven have partnered to provide a full-spectrum, unified Kafka solution for real-time applications in highly regulated environments.