December 2015 Summaries

The concept of the Lambda Architecture is centered around modeling complex computing systems as an ordered, immutable log of events. This approach allows for processing data as a series of transformations that output to new tables or streams, with each piece being independent and having defined inputs and outputs. The key property of this system is its ability to replay parts of the dataflow, making it easier to test and parallelize. What sets Lambda apart from other systems is its technique of writing data to two places, optimizing for both space and time, which allows existing map/reduce systems to be upgraded with a new fast track. The architecture itself isn't unique, but rather a sensible set of data engineering practices wrapped up under a catchy rubric, which can help people add low-latency fast tracks to existing big data systems.

The Lambda Architecture is a design pattern used to manage real-time and historical data concurrently. It forks into two paths: one for streaming (real-time) data and the other for batch data. Many Internet-scale companies, such as Pinterest, Zynga, Akamai, and Comcast, have chosen SingleStore to deliver the high-speed data component of the Lambda architecture. SingleStore can fulfill both sides of the Lambda architecture, supporting real-time streaming and batch services. It delivers real-time analytics on a rapidly changing data set, making it an ideal match for the characteristics of the Lambda Architecture speed service. SingleStore offers scalability through its distributed shared nothing architecture, supports multi-model, multi-mode data ingestion, and provides full ANSI SQL support, including JSON support. Additionally, SingleStore has open source connectors for smooth integration with other data sources, such as Apache Spark and Amazon S3. Customers from various industries have invested in SingleStore to power mission-critical applications, driven by the value of real-time data analysis and the power of SQL.

The industrial internet is expected to move towards real-time data pipelines, enabling businesses to detect patterns and adjust operations in near real time, with the Internet of Things playing a key role. This will have a significant impact on industries such as energy, pharmaceuticals, and agriculture. Consumer visibility into business operations will become more granular, allowing consumers to make informed decisions and build brand loyalty. The cost of doing business is expected to decline due to advancements in storage technology and the increasing accessibility of commodity hardware. Additionally, the crowdsourcing of analytics is becoming increasingly popular, enabling people from around the world to participate in data analysis and generating insights through human-in-the-loop processes.

This release of SingleStore Ops includes performance features for Streamliner, their integrated Apache Spark solution. The latest release allows users to run Spark SQL inside the SingleStore database, enabling faster execution with improved NUMA-aware deployments and in-browser Python programming capabilities. By leveraging PySpark and a `pip` command, users can now prototype data pipelines much faster using Python syntax, which is widely used by developers. This focus on Python aligns with SingleStore's existing software written outside of the core database, allowing generalist developers to get up and running quickly. The release also includes performance boosts from running Spark SQL in SingleStore, where the optimized engine processes individual queries much faster than Spark can.

Many legacy database systems are not equipped for modern applications that require high-velocity, high-volume data workloads, driven by near ubiquitous connectivity. To support such workloads, database systems must have characteristics like real-time ingestion and processing, subsecond response times, anomaly detection as events occur, and the ability to generate reports over changing datasets. Companies are using in-memory solutions to meet these requirements, with examples including Pinterest's real-time analytics pipeline and Novus's portfolio management platform, which provide instant answers to analysts querying their dataset. As more data comes online, organizations will need systems that can rapidly ingest data while simultaneously making it accessible for analysis, highlighting the importance of building real-time data pipelines through in-memory architectures.

SingleStore is a distributed relational database that can be used as an alternative to Oracle in certain scenarios, offering advantages such as faster data processing and analytics capabilities. It has two primary data stores: an in-memory rowstore and a disk-based columnstore, which allows for real-time analytics and stream processing. SingleStore complements Oracle databases by serving as the real-time analytics engine, ingest layer, or stream processing layer, eliminating the need for batch processing and enhancing enterprises' ability to make decisions in real time. With its unique architecture and benefits of scale out, distributed processing, and memory-first technology, SingleStore is an emerging database vendor that can be considered by enterprises with existing Oracle databases.