February 2017 Summaries

Building a real-time application requires connecting the pieces of your data pipeline to ingest, transform, store and make data easily accessible at sub-second speeds. A typical architecture consists of an ingestion layer that captures feeds, a transformation tier that distills information and delivers formats, a storage layer for persistence and analytics, and querying capabilities using SQL to power real-time dashboards. As new applications generate increased data complexity and volume, it's essential to build an infrastructure for fast data analysis to enable benefits like real-time dashboards, predictive analytics, and machine learning. Industry leaders are sharing their experiences on building ideal technology stacks for real-time analytics, such as Pinterest and an energy company that used Kafka and SingleStore to monitor sensor data and reduce risk of drill bit breakage.

Myntra's new processing and reporting framework, called Meterial, utilizes Apache Kafka, Data transformer based on Apache Spark streaming, SingleStore real-time database, and a UI built on React.js to provide real-time insights into rapidly growing data. The framework is designed to process unstructured/semi-structured data in real time with minimal latency, generating meaningful insights for business decisions. Meterial enables real-time user behavior analysis, traffic metrics, notification performance, and other key metrics, which are displayed through a dashboarding and reporting framework built on UDP APIs. The framework also supports ad-hoc analysis of commerce data with user data every 15 minutes, and is currently being explored for use in A/B testing and segmentation platforms.

SingleStoreDB Self-Managed 5.7 introduces a new pipeline extractor for Amazon Simple Storage Service (S3), enabling modern enterprise data lakes by harnessing the unlimited nature of S3 for ad-hoc and real-time analytics. A SingleStore S3 Pipeline extracts, transforms, and loads data from S3 objects in parallel to each database partition, achieving exactly-once semantics and sub-second performance for streaming data queries. With SingleStore Pipelines, database developers can easily create, test, alter, start, stop, and configure pipelines using basic DDL statements or a GUI, allowing for rapid and continuous data ingest for real-time analytics with ANSI SQL and business intelligence applications. To get started, users need to open an AWS account, download a 30-day free trial of the SingleStore Enterprise Edition, and create their first pipeline using the S3 Pipelines Quickstart guide.

The growth of data has led to an increased impact on analytics, which is critical for delivering a successful digital business strategy. Companies are using more real-time analytics due to the pressure to speed up and improve business processes, particularly in digital businesses and the Internet of Things (IoT). A well-designed real-time system must process data quickly, considering technologies such as in-memory databases, parallel processing, efficient algorithms, and innovative data architectures. To ensure a proper investment return, the response time of analytics must align with the speed of the business decision, taking into account factors like value degradation and decision improvement. Additionally, automating decisions using algorithms requires a well-defined process to code against, as per Gartner's recommendations.

SingleStore is releasing a new Spark connector that will operationalize powerful advanced analytics, showcasing its capabilities at Spark Summit East 2017. The event will feature talks by SingleStore's CTO and Co-founder Nikita Shamgunov and product manager Steven Camiña, including sessions on architecting real-time data pipelines with Apache Spark and building the ideal stack for real-time analytics. Attendees can also expect to learn from industry experts at various other sessions, as well as participate in a happy hour hosted by SingleStore. The event is scheduled for February 7-9 at the John B. Hynes Convention Center in Boston, MA.