January 2017 Summaries

Microservices architecture is gaining popularity due to the need for real-time applications that require deep analytics and scalable components. This trend is driven by the increase in real-time applications, which necessitate a strategy for manageable components delivering speed and scale. Microservices allows developers to focus on building business logic rather than coordinating with corporate models and infrastructure constraints. However, it also poses challenges such as distributed transactions, eventual consistency, and high performance analytics. SingleStore can help address these challenges by providing a flexible data repository that delivers high performance joins, immediate consistency, and rich analytics capabilities. By leveraging SingleStore, developers can create a highly flexible, scalable, and performant solution that addresses the complexities of Microservices architecture.

Manage, a technology company specializing in programmatic mobile marketing and advertising, generates over a terabyte of data daily and processes more than 30 billion bid requests. To manage this scale, they initially used MySQL but later switched to Hadoop with Apache Hive and Kafka, only to find that Hive was slow. In search of a faster solution, Manage turned to SingleStore, which enabled them to reduce the delay in data freshness from two hours to 10-15 minutes, allowing for real-time analytics and ad-hoc queries on log-level data within seconds. With SingleStore Streamliner, an Apache Spark solution, they stream log data from Kafka, store it in a columnstore, and aggregate it into summary tables, providing a highly scalable and real-time data pipeline.

We at SingleStore love hosting meetups at our headquarters, which provide a hands-on way to share our work, connect with the community, and get in-person feedback. Last year's meetups covered topics such as building real-time data pipelines and powering geospatial dashboards, including sessions led by experts like Chandan Joarder from Macy's and Matt Irwin of Mapbox. These events showcased innovative approaches to data exploration and visualization, demonstrated the use of Apache Kafka with Confluent and SingleStore, and highlighted the importance of exactly-once semantics in real-time streaming applications.

The concept of the S-Curve is widely applicable in various industries, including business and technology. Long-term success relies on adaptation and reinvention, particularly in a dynamic world where nothing lasts forever. The database ecosystem is undergoing significant changes due to megatrends such as social media, mobile devices, cloud computing, big data analytics, IoT, machine learning, and the need for constant innovation to stay ahead of the competition. The S-Curve concept explains the evolution of successful new technologies or products, where early adopters provide initial momentum, followed by a steep ascent as more people join in, and finally, the curve levels off sharply as adoption approaches saturation. Companies that successfully scale multiple S-curves are characterized by identifying substantial market changes, possessing threshold competence before scaling up, and attracting and retaining high-performance talent. The S-Curve also applies to the evolving database and data management world, where distributed systems have emerged to power a new era of business progress, providing advantages such as scalability, performance, alignment with CPU trends, economic efficiencies, and deployment flexibility. As AI continues to augment datastores, we can expect to see natural language queries, efficient data storage, pattern recognition, and more. To jump the Database S-Curve, companies must be creative in discovering new insights and talent, adopting new products and technologies, and embracing change to stay ahead of the competition and achieve long-term success.

The rapid growth and impact of machine learning in 2017 will be significant, with companies using real-time technologies to power applications, driven by emerging data sources and trends such as energy companies' use of predictive analytics in wind farms and oil fields. The podcast discussion highlights the importance of building a real-time data pipeline, considering deployment considerations, and managing teams specializing in machine learning, while also exploring the overlap between machine learning and data science fields. A new book, "The Path to Predictive Analytics and Machine Learning", provides a playbook for building applications that take advantage of predictive analytics and machine learning, offering insights into the latest step in real-time analytics journey.

In today's digital landscape, protecting sensitive data from cyber attacks is crucial for the success of large government agencies and commercial companies. SingleStore has a comprehensive security focus, including protection against insider threats, and offers best practices for securing data infrastructure at the database tier. The primary goal of separating administrative duties is to disintermediate the Database Administrator (DBA) from sensitive data, allowing only approved administrators to grant themselves privileges without approval by a second administrator. This can be achieved through setting up recommended roles, such as Compliance Officer and Security Officer, which manage role permissions and activity at both the organization and project levels. Additionally, SingleStore's Strict Mode configuration ensures that the security environment is permanently locked down once deployed, preventing rogue administrators from modifying configurations in production systems.