January 2016 Summaries

The Hive think tank recently held a panel discussion on Stream Processing Systems, covering various topics including messaging platforms, scalability, latency, fault-tolerance, deployment simplicity, maturity, enterprise features, and security. The conversation was moderated by Ben Lorica, Chief Data Scientist at O'Reilly Media, featuring experts from prominent companies such as Twitter, MapR, SingleStore, Hortonworks, and Confluent. Panelists discussed improvements in Spark streaming compared to Storm, the reasoning behind Twitter's custom messaging system, customer concerns about latency and streaming, changes in Kafka, upcoming trends in the stream processing space, and strategies for preventing specific data access to consumers. The Hive Think Tank is an incubation studio focused on data-driven applications, bringing together a community of over 8,000 data experts to launch and support startups.

Modern businesses are adopting the Lambda Architecture to support an increasing variety of data workloads and uses cases, which require both fault-tolerance and scalability. The architecture models everything as an ordered, immutable log of events, with processing completed as a series of transformations that output to new tables or streams. SingleStore combines database and data warehouse workloads, enabling both transactional processing and analytics, often fulfilling the speed layer of the Lambda Architecture by providing in-memory performance. To make sense of the architecture, a new guide is being launched, The Lambda Architecture Simplified, which demystifies complexity surrounding the model and provides simplified data frameworks and real-world use cases.

The author created a custom workload generator called `dbbench` to drive an investigation into a Linux performance issue affecting a specific customer workload running on SingleStore, a NoSQL database. The issue was caused by the third-party benchmarking tool inadvertently throttling itself when running against SingleStore, leading to incorrect results. The author used `dbbench` to simulate a complicated analytics workload and observed interesting performance behaviors, such as how read query performance changes dramatically after table cleanup. By monitoring and questioning the data throughout the execution of the benchmark, the author was able to identify and fix a complicated performance anomaly. `dbbench` is now publicly available under an Apache license and is being actively used by SingleStore for customer Proof of Concepts and performance testing.

The SingleStore meetup series has been hosting regular meetups in San Francisco's SoMa neighborhood, aiming to connect with tech startups and share knowledge on various topics including real-time digital insight, database performance, scaling startups, and more. The first official meetup of 2016 was recently held, showcasing the progress made over the past year. Key speakers included Carlos Bueno, Jim Herbold, Drew Paroski, Chris Fry, Ciprian Gerea, and Eric Boutin, who shared their expertise on topics such as in-memory database performance, scaling hypergrowth sales, writing compilers in modern C++, rapid startup scaling, lessons learned from building real-time databases, and Microsoft's Cosmos for business intelligence. The meetups have covered various aspects of technology, providing valuable insights to attendees and promoting knowledge sharing within the tech community.

The Internet of Things (IoT) is expected to have a real-time database behind every sensor, with billions to trillions of devices predicted. This will lead to an increasing number of new data streams, making instant analytics on real-time data crucial for digital transformation. The infrastructure to support this includes databases that can ingest, analyze, and store massive amounts of data in real-time, providing a system of record while simultaneously offering real-time analytics. These databases must also be able to respond to and integrate well with familiar ecosystems, scale online, and provide enterprise-level service quality. With immediate access to both real-time and historical data, companies can shape customer experiences and accelerate applications for wearables, drones, and vehicles in the moment.

The author, a performance engineer at SingleStore, encountered an uncommon Linux performance issue while running a synthetic workload on a columnstore table. The workload was using 16 threads to execute a simple query against the table, but the CPU cores were spending about 50% of their time idle. Using `perf_events` and an `awk` script, the author created an off-cpu flamegraph that revealed that every `mmap` syscall was taking around 10-20ms due to contention on the `mm->mmap_sem` lock. The issue was caused by SingleStore's use of `mmap`, which was inadvertently benchmarking Linux's `mmap` functionality, leading to significant performance overhead. To resolve the issue, the author switched from using `mmap` to the traditional file `read` interface, resulting in nearly doubled throughput and becoming CPU bound as expected.