May 2017 Summaries

To become a responsive data-driven business, organizations must address current data latency challenges that can hinder contextual user experiences, revenue identification, and expenditure prevention. These challenges encompass slow data loading, lengthy query execution, and low concurrency issues. A real-time data warehouse solution, such as SingleStore on Cisco UCS Integrated Infrastructure for Big Data and Analytics, provides a scalable platform to deliver fast, accurate, secure, and always available data. This enables high-performance applications to require fast, accurate, secure, and always available data, scaling linearly to millions of events per second while analyzing petabytes of data for insights.

The Spark Summit 2017 conference is about to take place, featuring over 175 talks led by top experts in the Apache Spark ecosystem. These sessions will cover various topics such as machine learning, data science, and real-world case studies, aiming to enhance attendees' skills in these areas. The conference includes a range of informative talks on machine learning, including discussions on the trajectory of MLlib, embracing a taxonomy of types for simplification, extending Spark's machine learning pipelines, and applying AI to increase user retention. Additionally, there will be live demos and a guide available for designing, building, and deploying Spark applications using the SingleStore Spark Connector.

SingleStore is a leader in the real-time analytics space, used by leading companies such as Comcast, Akamai, and Samsung to drive value from data with real-time data warehousing and analytics. To help enterprises navigate the rapid change in the digital business landscape, SingleStore partners with industry innovators like Twingo, Tableau Software, and Zoomdata to design, build, embed, and implement real-time enterprise solutions. The SingleStore Partner Program provides partners with resources, expertise, and technical support to thrive in the data-driven economy, enabling them to bring innovative real-time data warehousing solutions to enterprises quickly worldwide.

The traditional data warehouse solutions, designed for the requirements of 20 years ago, have become outdated due to explosive data growth and new workloads. These solutions were expensive, inflexible, and struggled with scalability, batch ingestion, and ease of use. The introduction of cloud and distributed systems has provided a strategic shift towards agility and cost optimization. SingleStore is a real-time data warehouse optimized for hybrid cloud deployments, exceling at operational use cases, offering features such as high-performance SQL, reduced maintenance costs, flexible deployment across clouds and hardware types, and scalable distributed architecture. With SingleStore, users can rescue themselves from their dated systems and take advantage of its benefits to perform faster queries, reduce maintenance costs, run anywhere with a flexible software footprint, analyze all data formats, and ensure enterprise-grade security across on-premises and cloud deployments.

In the traditional data warehouse era, enterprises relied on batch processes to generate analyses and reports, utilizing extraction, transformation, and load (ETL) processes to standardize disparate data into normalized schemas. In contrast, today's real-time economy demands perpetual data ingest, simultaneous reads, high user concurrency, and fast queries, making it necessary for businesses to shift towards real-time data storage and analytics. Companies like Pinterest, Uber, and Pandora have achieved significant performance advantages by adopting this approach, as stated by Eric Frenkiel, SingleStore co-founder and CEO. The real-time data warehouse is designed to continually load and transform data, enabling applications to run on top of it with exactly-once semantics. With a hybrid cloud data warehouse like SingleStore, businesses can push real-time workloads where they are most economical to run, offering cost-savings advantages in the process.

SingleStoreDB Self-Managed 6 Beta 1 has introduced a new feature called Procedural SQL, which supports the creation of user-defined functions (UDFs), stored procedures (SPs), table-valued functions (TVFs), and user-defined aggregate functions (UDAFs). This allows for more convenient programming with arrays, which can be declared and initialized using simple language constructs such as `declare a array(int) = create_array(10);`. Arrays in SingleStoreDB Self-Managed 6 Beta 1 are zero-based, meaning the positions of the array are numbered starting from 0. The compiler uses type inference to determine the data type of the array based on the variable being assigned to it. This feature enables performance-optimized code generation and can sort large arrays quickly. Multidimensional arrays with "ragged" sub-arrays are also supported, allowing for more complex data structures. Overall, arrays in SingleStoreDB Self-Managed 6 Beta 1 provide a powerful toolset for developers to create efficient and expressive code.

The SingleStore Spark Connector is a powerful tool that enables users to leverage the capabilities of Apache Spark in conjunction with the fast data ingest and durable storage benefits of SingleStore. By connecting Spark workers directly with SingleStore partitions, it allows for parallel read and write operations, improving write performance and enabling real-time data ingestion. The connector also supports "SQL Pushdown", which automatically translates Spark SQL queries into SingleStore commands, further enhancing efficiency. With its simple and lightweight API, users can easily prepare, execute, and persist Spark DataFrames in SingleStore using methods such as `SingleStoreContext.sql()` and `df.saveToSingleStore()`. The connector is designed to be used with Apache Spark for transforming large datasets and storing data in a persistent and efficient format.

SingleStore is an Enterprise Cloud database that delivers the requirements for a multi-tenant platform through its rich analytical and transactional capabilities, scalability to hundreds of nodes in a cluster leveraging shared nothing commodity hardware, and extreme data ingestion. The key to implementing multi-tenancy is selecting one of three architectural patterns: Separated Database, Separate Schema, or Shared Schema. Each pattern has its benefits and drawbacks, such as increased maintenance costs for the Separated Database model, cost savings with reduced hardware resources in the Separate Schema model, and painless schema changes with the Shared Schema model. SingleStore supports these models through features like Role-Based Access Control (RBAC) and Row-Level Security (RLS), allowing developers to choose the best approach for their multi-tenant application needs.

At Strata+Hadoop World, SingleStore Software Engineer John Bowler shared two ways of making production data pipelines in SingleStore, one using Spark for general purpose computation through a transform defined in SingleStore pipeline. He ran a live demonstration of SingleStore and Apache Spark for entity resolution and fraud detection across a large dataset, leveraging SingleStore's native geospatial capabilities to reduce network overhead. John used SingleStore Pipelines and TensorFlow to write a machine learning Python script that accurately identified handwritten numbers after training the model in seconds, showcasing the performance benefits of combining SingleStore with popular open-source libraries like Duke for entity resolution. The presentation provided a 79-page guide on designing, building, and deploying Spark applications using the SingleStore Spark Connector, along with code samples and performance recommendations for production-ready Apache Spark and SingleStore implementations.

The world's most valuable companies, including Apple, Google, Microsoft, Amazon, and Facebook, are harnessing data to drive their success. These companies share a common approach to analytics that involves capturing and processing large amounts of data from various sources, including app stores, online music, video, and books. The app store example illustrates the characteristics of these platforms, which include hundreds of millions of end users, millions of application developers, dozens of app segments, and one primary platform provider. To meet the needs for comprehensive analytics across this stack, data solutions need to provide fast data capture, low latency query capability, high concurrency, and analytical capabilities that support real-time analytics, ad hoc analytics, and analytics for app developers and end users. A multilevel approach, self-service analytics, and a scale-out distributed system with modern query execution systems are recommended strategies for achieving analytics at scale in these industries.