November 2015 Summaries

I became disillusioned with the sales engineering role after realizing that many operations people spent their time putting out fires instead of automating and improving. I joined Atlassian in 2008, where software development was undergoing a sea change in the right direction, focusing on Agile and Git. I'm now excited to join SingleStore, where companies like Google, Facebook, and Netflix prioritize handling massive data sets to make real-time decisions that delight customers. The goal is to build better, stronger, and faster applications with the Community Edition database, which can handle anything thrown at it.

In today's fast-paced digital landscape, organizations that can process data quickly gain a significant competitive edge. To help businesses make informed decisions in real-time, experts from SingleStore and O'Reilly Media teamed up to host a webcast on building efficient data pipelines. The discussion covered ideal technology stacks, simplifying Lambda architectures, utilizing memory-optimized technologies like Kafka and Spark, and sharing examples of companies like Pinterest and Comcast that have successfully implemented real-time workloads. By leveraging these strategies, businesses can reduce data latency, simplify their data architectures, and unlock valuable insights from their data. With the rapid pace of technological advancements, staying ahead of the curve is crucial for organizations looking to remain competitive in today's digital economy.

The financial markets have experienced extreme volatility in recent months, leading to a high demand for real-time data processing and analytics. Traditional SQL database technology has struggled to keep up with the speed of financial markets, but modern systems like SingleStore are catching up. A script called gen.py simulates a distribution of bid and ask quotes for fluctuating stock prices, while another script called regress.py performs linear regression on recent quotes to estimate a stock's trend. With SingleStore, users can ingest real-time data, run queries simultaneously, and access historical data through a single interface, simplifying the process of building trading applications that model historical data and detect real-time market changes. The system allows developers to focus on actual analytics rather than developing complex infrastructure, making it easier for non-experts to leverage its power.

Streamliner is an open source tool that allows developers to build real-time data pipelines using Apache Spark, integrating data streaming from various sources into SingleStore for persistence and application serving. With minimal code required, users can instantly create their pipelines, performing data transformations and loading data directly within the Ops web UI or by uploading custom Extractor and Transformer classes. Streamliner supports complex operations on data, enabling real-time manipulation of data before inserting it into SingleStore. The tool is great for developers and data scientists looking to easily pre-process data while leveraging Spark's power, with examples and documentation available in the Streamliner repository and SingleStore documentation.