August 2017 Summaries

A new class of modern transactions has emerged, driven by IoT sensor information, website traffic logs, and global financial reporting, which require real-time data warehouses to handle the volume and velocity of data. These warehouses need to ingest and persist data in real time while serving low-latency analytic queries to a large number of simultaneous users. Incorporating machine learning capabilities into these warehouses can simplify data architectures and provide access to real-time information for faster critical decisions, enabling organizations to harness insights from vast arrays of live inputs.

Kafka Summit San Francisco is an event that brings together thousands of companies using Apache Kafka to share their experiences and learn from experts. The event features breakout sessions, lectures, demonstrations, and guest speakers covering technical content, customer stories, and new launch announcements. SingleStore will be exhibiting at the Sponsor Expo with a demo and subject matter expert availability for attendees to speak with. Recommended sessions include presentations on efficient schemas in motion with Kafka and Schema Registry, stream processing for everyone, real-time stream processing at scale, and best practices for using Kafka Connect.

We regularly host meetups at our headquarters to share new developments, connect with the community, and gather in-person feedback from attendees. We also invite partners and customers to join us for these events, which recently featured a meetup with Zoomdata. During this event, we shared two presentations on utilizing SingleStore and Zoomdata for real-time streaming, analytics, and visualization. The first presentation covered SingleStore system design, implementation, use cases, and leveraging Zoomdata for visualization. The second presentation discussed how traditional analytics tools lack streaming query capabilities and how modern solutions like Zoomdata can address these limitations, enabling fast analysis on streaming sources such as SingleStore. Both presentations are available to watch in their entirety.

SingleStore is a purpose-built data management system designed for high-volume workloads and provides a fast and secure platform for mission-critical analytics, offering real-time ingest, transform, model, and persistent storage of streaming content, with features such as separation of duties, full audit logging, database vault functionality, user authentication, encryption, enforced password protection, highly available data through level 2 redundancy and cluster replication. The system is capable of achieving high performance at large scale, with query speeds ranging in hundreds of millions to over a billion data points per second per core on commodity hardware, making it suitable for applications requiring real-time analysis of structured content such as telemetry data from drones or other IoT devices, while ensuring confidentiality, integrity, and availability of the data.

The NYC taxi data set is a large dataset of yellow taxi trip records from New York City, totaling over 1.3 billion rows. The data has undergone several schema changes over eight years, requiring careful handling and processing to load into a database. SingleStore makes it easy to load the data quickly and efficiently by using its native pipelines feature, which can process compressed files in parallel. The pipelines are designed to handle the large dataset and various file sizes, reducing the time required for loading and improving overall efficiency. Once loaded, the data can be analyzed and queried using geospatial queries, enabling insights into taxi trip patterns and behavior.