November 2018 Summaries

Aiven has integrated Prometheus, a popular open-source monitoring and alerting toolkit, into its platform. Users can now send their Kafka, PostgreSQL, Elasticsearch/OpenSearch, and Redis metrics to their Prometheus monitoring and alerting servers. Prometheus is modeled after Google's Borgmon and has quickly become the most popular Open Source monitoring and alerting tool available due to its active community and features. Its query language allows users to build complex alerts based on a number of factors, while its integrations enable it to receive and send metrics from various sources such as databases, visualization tools, and messaging systems. Prometheus is best used in combination with components like TimescaleDB for long-term storage and Grafana for visualization.

The future of data pipelines will be characterized by handling massive scale. As technology evolves, the amount of generated data grows exponentially, transforming in terms of amount, velocity, purpose, trajectory, and format. Data pipelines must adapt to accommodate this growth, focusing on functionality, design, compliance, usability, performance, and scalability. Future pipelines will need to support core-to-endpoint systems, handle near real-time data, and be capable of auto-scaling, sharding, and partition tolerance with minimal human interaction. They should also be troubleshootable and configurable on the fly, agnostic to various formats, and implement measures for error handling. Analytics pipelines will increasingly serve as a funnel/conduit for data used in training AI and ML models. Design considerations include incorporating a kill switch, making data in the pipeline modellable on the fly, and implementing immutable, ordered event logs. Compliance with regulations like GDPR is crucial, and various methods can be employed to secure data in an event log. Usability will involve GUIs for the entire lifecycle of pipelines, while performance improvements will focus on reducing latency and increasing mean-time-to-failure. Scalability will require decisions about which data to keep volatile and temporary and which to store persistently, as well as massive autoscaling capabilities.

Aiven has launched its seventh service product, Aiven for Apache Cassandra®, which is now generally available. This managed service is ideal for use cases with large, active data sets that need real-time querying or later analysis. It can be used in IoT applications and as a data archive for later analysis. Offering standard plan sizes that can scale to 9-node clusters supported by custom plans for even greater workloads, Aiven for Cassandra users will be able to address heavy workloads requiring tunable performance and consistency.

PostgreSQL 11 is now available on the Aiven platform, offering improved handling of large-scale data. Key updates include enhanced partition elimination, parallel query improvements, and expanded Just-in-Time compilation support. These enhancements aim to increase speed and resource management for users with large datasets.