September 2016 Summaries

SingleStoreDB Self-Managed 5.5 has been launched, featuring SingleStore Pipelines, a new way to achieve maximum performance for real-time data ingestion at scale, enabling exactly-once semantics when streaming from message brokers like Apache Kafka, thus delivering an end-to-end solution for real-time analytics and data ingest. This implementation supports both real-time analytical queries and real-time data ingestion, with SingleStore Pipelines ingesting data at scale in three steps: extract, transform, and load, ensuring atomic commitment of streaming data and exactly-once semantics through metadata storage. The new database object, pipeline, is a top-level element similar to table, index, or view, allowing extraction from data sources using a robust mechanism, enabling true exactly-once semantics for Kafka messages, and supporting data enrichment and transformation in various programming languages, with efficient and parallel data loading into SingleStore data partitions and Kafka brokers.

Oracle OpenWorld 2016 is taking place in San Francisco from September 18th with ten tracks, including a Data Center track focused on innovation in databases such as SingleStore and Oracle. SingleStore was designed to be a flexible ecosystem technology, offering users flexible deployments, open source connector tools, and a memory-first engine for real-time data ingest and analytics. This technology complements Oracle by serving as the real-time analytics engine, stream processing layer, or high-speed ingest layer, allowing customers to enhance scalability, distributed processing, and real-time analytics when combined with traditional systems such as Oracle databases. SingleStore will be showcased at booth 1821 during the event, offering attendees a chance to learn more about its capabilities and win prizes.

The Internet of Things (IoT) generates massive amounts of data daily, which can be analyzed in real-time to help businesses address consumer demands. Supply chain management is a key example of IoT's impact on manufacturing industries, where advanced methods for ingesting and analyzing IoT data are necessary due to the complexity of warehouse operations. Companies must adapt their strategies to include predictive analytics, as descriptive analytics alone is no longer sufficient. MemEx, a showcase application developed by Gartner, demonstrates how enterprises can implement predictive analytics to improve logistical operations. By harnessing sensor data from production lines and conveyor belts, supply chain organizations can predict throughput and plan accordingly. MemEx's architecture incorporates advanced technologies such as Apache Spark, SingleStore database platform, and JavaScript-based User Interface, enabling real-time analysis and prediction of warehouse states. The application enables enterprises to react to their businesses in real time, make supply adjustments, and drive revenue by predicting risk of production failures. As the future of supply chain management depends on predictive analytics, MemEx provides a realistic view into the capabilities of using real-time applications with predictive analytics.

The Path to Predictive Analytics and Machine Learning` is a book that explores the latest step in real-time analytics: predictive analytics, and provides a playbook for building applications that take advantage of machine learning. The book covers various aspects of real-time data processing, including building real-time pipelines, processing transactions and analytics in a single database, and applying existing batch processes to real-time data pipelines. It also introduces the reader to applied machine learning, highlighting tradeoffs between training and scoring latency, bias and variance, and accuracy versus complexity. The book aims to make predictive analytics more accessible by combining well-understood machine learning techniques with technology advances in software and hardware, ultimately paving a logical path forward for making the leap from machine learning to broader artificial intelligence.

Apache Kafka is a high-throughput messaging system used to capture data at its source and power real-time analytics. It provides ideal performance for real-time pipelines by scaling producers and consumers through distributed architecture, combined with commit log on disk. Data from Kafka is then persisted to a memory-optimized database like SingleStore, which rapidly ingests and transforms the data for users to build applications on top of. The collaboration between Apache Kafka and SingleStore enables companies to deliver instant answers at scale by architecting real-time systems that anticipate customer needs and fulfill expectations for fast, personalized services.