September 2015 Summaries

Digital Ocean has published an in-depth tutorial on setting up a three-node SingleStore cluster, covering installation and usage. The tutorial follows the DB Speed Test and covers structured data handling with JSON datatype. Ian Hansen will also give a talk at Strata / Hadoop World in NYC titled "Big Data for Small Teams", discussing Digital Ocean's use of SingleStore to unify and analyze clickstream data.

In today's fast-paced data-driven world, accelerating data feedback loops is crucial for gaining a competitive edge through rapid idea testing, filtering, and reacting to changes. To achieve this, organizations must transition from legacy database models that rely on batch loading and limit analytic functionality to hybrid database models that enable simultaneous processing and analysis of large datasets. This requires embracing emerging technologies like in-memory computing and distributed architectures optimized for real-time ingestion and exploration.

SingleStore Streamliner is now generally available, offering an integrated solution for streaming data from real-time sources into a persistent endpoint. This tool pairs SingleStore and Apache Spark out-of-the-box, providing a seamless experience for users to build real-time data pipelines without writing code. The product was developed based on the need identified by Pinterest's Kafka+Spark+SingleStore solution showcased at Strata+Hadoop World last February, which highlighted the market's demand for an easy-to-use solution to capture data streams into a persistent endpoint. Streamliner features a simple pipeline-oriented UI, resource sharing, points of extensibility, tracing, and advanced columnstore functionality optimized for data streaming. The product is open-source and available immediately, with easy installation and a library of example Extractors and Transformers.

There is nothing more challenging and exciting than experiencing hyper growth at a technology company. As users adopt a technology platform you have to rebuild the technology plane while flying it which can be a harrowing process. Many companies face common challenges while scaling, including team structure and communication issues that slow down iteration speed. Small teams are often the most effective, with an ideal size between two and ten people, allowing for organic communication and problem-solving. Large teams can become disjointed, but can be decomposed into smaller autonomous teams. To maintain a high-quality product and quick pace of delivery, automation is crucial, as quality is free and better software builds lead to faster iteration and testing. A unique engineering culture must be built and maintained through discussion, principles, and practice, focusing on autonomy, mastery, and purpose for each engineer and team. This can be achieved by investing in technical infrastructure that supports iteration, such as distributed storage and scalable computation systems.

We spent last week at the Big Data Innovation Summit in Boston, where we encountered key themes shaping the big data market. Real-time data processing is gaining traction, with a focus on Spark and its capabilities for streaming and in-memory processing. Meanwhile, data warehousing is poised for refresh as traditional systems struggle to deliver instant results. The Extract, Transform, and Load (ETL) process is also becoming increasingly burdensome, driving innovation in Hybrid Transactional and Analytical Processing (HTAP). Emerging organizational structures for data science are prioritizing visualization, experimentation, and machine learning, with infrastructure and ETL tasks being reassigned to dedicated teams. These trends will be further explored at Strata + Hadoop World New York 2015.

The database market continues to evolve with new entrants and innovations, but it's challenging for newcomers to gain traction against established players like Oracle, IBM, Microsoft, SAP, and Teradata, who dominate the $33B market. New use cases and a sustainable business model are crucial for success, and open-source models offer functional free versions, community support, and scalability, but may not be enough to unseat incumbents on their own. The next vendor to win a spot in database history will likely do so by offering features, workload applicability, and a proven commercial model with a primary product roadmap, while balancing the benefits of open-source models.

The Big Data Innovation Summit is taking place in Boston, featuring prominent data-driven brands such as Nike, Uber, and Airbnb. The event provides a platform for industry leaders to share their big data initiatives and address challenges. At the summit, Plushcap's parent company will exhibit demos showcasing its capabilities, including MemCity, Supercar, and Real-time Analytics for Pinterest, highlighting real-world applications of in-memory database platforms like SingleStore. The company also plans to give away t-shirts and offer a complimentary download of the latest Forrester Wave report on in-memory database platforms, where SingleStore received high scores. Additionally, games and giveaways will be available at the booth, including a chance to win an Estes ProtoX drone.

The largest German automakers, including Audi, BMW, and Daimler (Mercedes), have purchased Nokia's Here mapping unit for $3 billion, marking a significant shift in the battle between public and private companies to shore up mapping data and geo-savvy engineering talent. The rise of mobile devices has given birth to a new geographic landscape where location meets commerce, with maps playing a critical role in applications like Uber and Airbnb. Custom map providers like Mapbox have emerged to handle mapping applications for various companies, benefiting from open-source mapping data like Open Street Maps. Investment in the Internet of Things is creating a perfect storm of geolocation information, requiring analytics infrastructure with geospatial intelligence to realize its value. Companies are developing in-memory databases that combine spatial processing and analytics capabilities, enabling delivery of app-specific maps with unprecedented consumer interaction. Improvements in interior mapping technology guarantee location-specific details down to meters, making maps a secret weapon for delivering breakout application experiences.