April 2021 Summaries

Prometheus is a popular open-source monitoring and alerting application, but it may not be sufficient for all needs. M3, an extension for Prometheus, offers several advantages such as long-term data storage, faster reporting, comprehensive query results, high availability from the ground up, and scalability without overloading. Aiven provides a fully managed cloud solution for implementing M3, making it easier to adopt and use.

Developers are incorporating Apache Kafka® into their tech stacks to achieve event-driven systems. Designing effective data payloads is crucial for successful communication between different components in a system. This article provides tips on designing Apache Kafka data payloads, including using all features of Kafka records, organizing data structures, choosing appropriate data formats, and incorporating timestamps. It emphasizes the importance of only including necessary fields in payloads and considering the context and use cases of consumers when designing data structures.

Apache Kafka version 2.8 has been released with several enhancements, including a preview feature allowing Kafka to run without Zookeeper using the RAFT protocol (KIP-500). Starting from this version, Kafka producers will have "exactly once" delivery semantics enabled by default (KIP-679), and performance improvements include the ability to limit connection creation rate on brokers (KIP-612) and improved leader election performance for clusters with thousands of partitions. Aiven now supports Apache Kafka version 2.8, allowing users to either launch a new cluster or upgrade their existing one.

Multicloud refers to setting up a data architecture that uses multiple cloud environments and regions, rather than relying on just one. This approach has become increasingly popular, with 81% of public cloud users already using two or more providers. The benefits of multicloud include access to the best tools and features from different clouds, enhanced performance through increased bandwidth and storage, freedom from vendor lock-in, and improved backup capabilities. However, managing complexity, addressing security concerns, and maintaining loyalty bonuses can pose challenges for organizations adopting a multicloud strategy. A unified management system like Aiven's Teams feature can help manage multiple clouds more effectively, while also providing robust security measures and flexibility in terms of cloud providers.

Aiven has announced its intention to join the community forming around the new fork of OpenSearch and OpenSearch Dashboards, spearheaded by AWS. The company is eagerly awaiting the first iteration of the OpenSearch project and looks forward to being an active contributor. This move comes after Elastic.io's announcement of a change in their licensing policy that no longer permits Elasticsearch to be offered as a service to third parties. Aiven, along with other companies, took a stance for open source, leading to the upcoming open-source fork for Elasticsearch and Kibana. The new project includes OpenSearch (from Elasticsearch 7.10.2), OpenSearch Dashboards (from Kibana 7.10.2), and Open Distro for Elasticsearch (providing additional features). Everything will be released under the Apache Licence Version 2.0, free to modify, extend, embed, monetize, resell and offer as a service.

Open source adoption is expected to accelerate in 2021, with significant growth in cloud market and digital transformation. The global public cloud infrastructure market is predicted to grow by 35% this year, reaching $120 billion. Time series databases are anticipated to be in high demand as businesses increasingly rely on managed services for data management.

Companies in regulated industries often lag behind in cloud adoption due to concerns about data security compliance. However, maintaining certification with major data regulations and obtaining ISO 27001 certification can help cloud service providers (CSPs) convince regulatory bodies that customer data is safe throughout its lifecycle. Aiven, a CSP, maintains compliance in all major data regulations and is ISO 27001 certified to ensure secure storage of customers' data. Compliance with security standards benefits everyone involved in the cloud ecosystem.