January 2019 Summaries

In today's fast-paced business landscape, organizations rely heavily on data-driven decisions to gain a competitive edge. However, current databases often fall short in meeting these needs due to limitations such as data democratization, performance degradation, and expensive upgrades. SingleStore offers a converged architecture that provides extreme performance, massive scalability, an easy-to-use SQL interface, cloud-native capabilities, and reasonable pricing, making it an attractive solution for a wide range of use cases including financial services, media and communications companies, and more.

Time series data has seen significant growth in recent years, with increasing interest in its applications across various industries. The growing demand for time series data storage and analysis has led to the development of specialized databases designed to handle this type of data efficiently. A key database that stands out is SingleStore, a general-purpose NewSQL database that offers strong scalability, fast performance, and robust SQL support, making it suitable for a wide range of use cases, including real-time analytics, machine learning, and AI applications. While dedicated time series databases, such as Graphite, InfluxDB, OpenTSDB, and kdb+, offer specialized optimizations, they often come with limitations in terms of structure, query performance, and scalability. SingleStore's ability to seamlessly scale out for high-performance, memory-optimized storage, and fast ingest performance make it an attractive option for organizations looking to leverage time series data effectively.

Time series data has seen an increasing focus over the past decade, with interest more than doubling. It's at the core of the Internet of Things and involves frequent sensor readings from various devices. Time series data is inherently well-structured yet can be complex when it isn't. Traditional databases may struggle to process large volumes of time series data, but new technologies like SingleStore are designed to handle this challenge. The emergence of machine learning and AI has driven the importance of time series data, with companies seeking to analyze and act on vast amounts of data to gain insights and stay competitive. However, many organizations still isolate their time series data in NoSQL databases, hindering analysis and transactions, highlighting the need for a unified solution like SingleStore that offers both operational and analytics capabilities.

A Fortune 50 company struggled to meet its fast analytics needs due to increasing data volumes and user demands for real-time access. They initially tried traditional SQL databases, then Hadoop, but found both inadequate. The company finally achieved success by augmenting Hadoop with SingleStore, a NewSQL database that offers a cloud-native, distributed architecture, streaming ingest, transactions, and analytics, including SQL support. With SingleStore, the company can now provide direct access to live data for faster analytics performance, executive accessibility, and improved concurrency and query performance. The deployment process was successful, with a time-to-live of less than three quarters, managed by a single team, and offered superior total cost of ownership.

Storing and analyzing large amounts of data is no longer the primary focus for successful companies, as speed in providing relevant information to decision-makers has become more crucial. Streaming analytics help identify perishable insights, which require immediate attention to avoid missing business opportunities. However, many companies view implementing a streaming analytics platform as a complex and costly project. In reality, using the right technologies and tools can set up such a platform quickly and effectively. A solution that identifies fraudulent ATM transactions in real-time has been built using a simple architecture, leveraging Confluent Kafka for data buffering and KSQL for SQL-like querying capabilities. The high-level architecture is comprised of three steps: building and analyzing streams of data, ingesting streams into a data store in real-time, and visualizing the data in real-time. This solution utilizes SingleStore for data ingestion and storage, which integrates seamlessly with Confluent Kafka through SingleStore Pipelines. Zoomdata is used to visualize the data in real-time, leveraging its smart query engine and Data DVR technology to connect to the source data stream immediately, reflecting changes as they occur.

SingleStore is now free to use, with up to 128GB of RAM usage, allowing developers to build and test projects without incurring costs. The company's database is highly capable, with a tiered storage architecture that combines the benefits of memory-led and disk-based storage. SingleStore supports ANSI SQL, distributed storage, and enterprise security features. It fits well into existing data processing architectures and can be used with various tools for data ingest and analytics. A recent webinar demonstrated how to build an analytics app in just a few minutes using SingleStore, showcasing its ease of use and performance capabilities. The database is designed to work with popular business intelligence tools like Looker, and its architecture includes master nodes and leaf nodes that enable efficient query processing. With its free tier and flexible licensing options, SingleStore offers a cost-effective solution for developers and organizations looking to build fast, scalable analytics applications.

We recently ported our 30,000 lines of JavaScript code from Flow to TypeScript, as we sought to leverage statically and strongly typed JavaScript for better type safety, reduced runtime errors, and improved developer productivity. We found that using TypeScript provided us with a more robust set of features compared to Flow, such as better support for third-party dependencies, enhanced editor/IDE integrations, and improved performance. However, we also encountered some challenges during the porting process, including fixing syntax errors, type inference issues, and adjusting our code to accommodate TypeScript's stricter standards. Ultimately, after one engineer week of work, we successfully fixed all remaining type errors and have been using TypeScript since then.

SingleStore was founded by Nikita Shamgunov after he left Microsoft, where he worked on a scalable transactional database that never shipped. Nikita joined Facebook and realized the power of such a system, leading to the co-founding of SingleStore in 2013. The company initially started as an in-memory system, which allowed for high-value workloads and early adoption by analytics teams. However, Nikita notes that building a fully scalable and elastic transactional system took time, with significant technological advancements required. SingleStore's architecture is based on RAM as the storage of the future, but has evolved to support both disk and memory due to rapidly dropping costs for flash memory. The company prioritizes consistency, availability, and partition tolerance (CAP theorem) in its distributed database design, sacrificing some availability to guarantee consistency and partition tolerance. Nikita discusses key architectural decisions, such as converting from row-oriented data to column-oriented data, which is I/O-bound and not CPU-bound. Performance optimization involves three buckets: in-memory query optimization, network optimization, and compression and vectorization. SingleStore is a general-purpose database that supports general-purpose workloads requiring scale, with typical use cases including real-time dashboards, Web 2.0 and mobile applications, portfolio analysis, and real-time financial reporting. The company aims to invest in managed services and Kubernetes integration, strengthen its engine, and improve the developer experience, while also exploring AI and ML workloads.