November 2016 Summaries

Data movement in systems design remains an obstacle, with batch Extract, Transform, and Load (ETL) being a perennial challenge. The shift from batch to real-time data processing is gaining traction, requiring new topologies and messaging semantics to be successful in the enterprise. Robust database systems can eliminate ETL processes, while shifting to real-time workflows normalizes compute cycles, mitigates traffic surges, and provides timely, fresh data for business value. Enterprise streaming opportunities involve taking batch processing into real-time workflows, with continuous data motion enabling insights for critical applications like improving customer experiences. Distributed data movement topologies, such as multipoint to multipoint communication, offer linear scalability and performance. Messaging systems rely on semantics like At Most Once, At Least Once, and Exactly-Once to ensure data integrity and delivery. The future of data movement is underway, with distributed systems and robust messaging semantics transforming enterprise batch processes.

Kellogg, the world's leading cereal company, relies on customer logistics data to make informed decisions and improve efficiencies around shopping experiences. To achieve this, Kellogg sought an in-memory solution to improve data latency and concurrency. The company turned to SingleStore to speed up its 24-hour ETL process, reducing latency from 24 hours to minutes. By integrating SingleStore with AWS and Tableau, Kellogg was able to run visualizations directly on top of the database, improving analytics performance by 20x. This enabled hundreds of business users to concurrently access up-to-date data, increasing the profitability of customer logistics data. With this in-memory solution, Kellogg can now provide business value on each project and meet the demands of its business users for speed and agility.

Smart gas and electric meters produce massive amounts of data, which can be handled by a small SingleStore cluster. This database is capable of handling large volumes of data from leading utility enterprises, including 200,000 meter readings per second, billions of rows of data, and 72 billion meter reads in a year, reducing storage needs to 624GB. SingleStore provides a highly performant and affordable analytics platform for Advanced Metering Infrastructure projects, enabling utilities to meet their goals and support data scientists using predictive analytics. The database leverages the MySQL wire protocol, allowing organizations with existing skills and tools to deploy it without modification. By empowering rapid analytics on Smart Meter data, SingleStore enables utilities to improve grid volume predictions, reduce costs, and deliver better customer service.

Today's business decisions are about seconds, not minutes. As companies move to evidence-backed decision making and widespread data access, they face two challenges: making data easily available and making it actionable in real time. With a centralized database approach, raw data is now shared with everyone, empowering employees to dive into the data and discover new opportunities. Innovations in data visualization tools like Tableau have produced powerful, interactive dashboards that afford companies the opportunity to be data-driven. However, users still face issues such as slow query execution, concurrency limitations, and siloed data, which can hinder their ability to make timely decisions. To address these challenges, SingleStore has collaborated with Tableau to provide a seamless business intelligence solution, enabling users to reduce query latency from minutes to seconds, putting time back on the clock and dollars back in businesses' pockets.

Tableau Conference 2016 is taking place in Austin, Texas from November 7-11, offering data engineers and business intelligence professionals the opportunity to learn about utilizing data analytics and visualizations. SingleStore will be exhibiting its native connectivity with Tableau at the conference, showcasing a new application called SingleStore Springs that demonstrates live customer behavior by demographic across resort properties, visualized with a Tableau dashboard. The company will also present a session on leveraging Tableau's APIs to improve data preparation and connection to various data sources. Additionally, attendees can attend sessions such as "ETL as Easy as ABC", "Connecting to Gnarly Data Sources", and others, which cover topics like connecting to novel databases, IoT dashboards, building a culture of data-driven analytics, and more. The SingleStore booth at the conference will offer complimentary books and a chance to win an Estes Proto-X Drone.

The urgency for IT leaders to bring real-time analytics to their organizations is stronger than ever, as the ability to start with fresh data and combine streaming, transactional, and analytical workloads in a single system can revolutionize operations. Data architects must carefully consider streaming semantics when moving from batch to real time, weighing factors such as bandwidth conservation and duplicate record prevention. Different types of streaming semantics, including at most once, at least once, and exactly-once, offer varying trade-offs between data reliability and efficiency. Achieving exactly-once semantics is ideal for operational applications, while SingleStore Pipelines can simplify the process by ensuring real-time data ingestion at scale from message brokers like Apache Kafka.