February 2021 Summaries

This post discusses how to connect a Go application with PostgreSQL using the pgx library, which is a PostgreSQL-specific alternative to database/sql. It also introduces Squirrel, an SQL query builder for Go that provides a more elegant and manageable way of building SQL queries compared to hardcoded strings. The author uses open data from the Kepler space mission to demonstrate how to connect to a PostgreSQL database, run queries, and build fluid interfaces with Squirrel.

Choosing the best time series database (TSDB) involves considering factors such as data type, storage, scaling and clustering, reading speed requirements, system footprint size, and logging and monitoring capabilities. Time series databases are commonly used for IoT applications due to their ability to handle large volumes of data with simple structures. Key considerations include the impact of data type on processing methods, storage options like compression or downsampling, scalability and clustering features, reading speed requirements, system footprint size, and logging and monitoring capabilities. Some popular TSDBs include M3, InfluxDB, TimescaleDB, and Prometheus, each with its own strengths and weaknesses. Ultimately, the choice of a TSDB depends on an organization's specific needs and requirements.

Aiven has introduced the first managed open-source database of its kind, Aiven for M3, to meet the growing need for purpose-built time series datastores. The new service is designed to handle high-throughput, real-world use cases and includes three components: a coordinator, database, and aggregator. By offering M3 Aggregator as a separate service, users can independently scale their aggregation workloads and make them highly available. Aiven for M3 is now available for existing customers to launch alongside other services or through a 30-day trial for new users.

In this post, Lorna Mitchell demonstrates how to populate an empty PostgreSQL database with data about exoplanets from the Kepler mission using Aiven's cloud platform. The process involves creating a new PostgreSQL service on Aiven, generating a table structure from a CSV file using ddlgenerator, and importing the CSV data into the newly created table. Once the data is imported, users can explore and analyze it using SQL queries or connect it to other tools for further analysis.

This tutorial demonstrates how to improve PostgreSQL performance by setting up a multiregion service using Python and Pulumi with Aiven. The process involves isolating read-intensive workloads to replicas, which can be helpful for analytical workloads such as interactive dashboards that use a read replica as the data source for reporting. Using Aiven, you can quickly deploy managed PostgreSQL services on public clouds like AWS, GCP, and Azure. Pulumi enables developers to create, deploy, and manage infrastructure on any cloud using their favorite programming languages. The tutorial provides step-by-step instructions on setting up the environment, configuring project settings for Pulumi, deploying the service with Aiven, testing the setup, cleaning up resources, and understanding the Python code details involved in creating Aiven infrastructure using Pulumi and generating test data using Python.

PostgreSQL 13 introduces improvements to indexing, sorting, and vacuuming that make datasets cleaner and queries faster. Key enhancements include deduplication of B-tree indexes, parallel vacuuming of indexes, improved partitioning support, and incremental sorting. These features can be leveraged by upgrading to PostgreSQL 13 or using Aiven's managed service for Postgres.

Apache Kafka is moving away from its dependency on ZooKeeper due to scalability and duplication issues. This change will impact the way brokers communicate with each other and with Kafka as a whole. In the post-ZooKeeper world, Apache Kafka would independently take care of its configuration management through a Raft quorum formed by controller nodes. The new architecture will also enhance the creation and deletion of topics. For Aiven for Kafka users, the main difference in a ZooKeeper-free Kafka will be improved performance. The latest release takes some further steps in that direction, impacting the way brokers communicate.

On January 14th, Elastic.co announced a change in the licensing of Elasticsearch from Apache v2.0 / Elastic combination to a more restrictive combination of Elastic License and SSPL. This caused concern among companies due to the use of SSPL which requires sharing the entire service source code freely at no charge if used as a service for third parties. In response, multiple companies began discussing alternatives. AWS announced on January 21st that they would be forking Elasticsearch and Kibana, forming a new open-source project with its own name to distinguish it from the original Elasticsearch project. The new project will have a governance model separate from any single company, ensuring contributions from the wider community. Aiven, an advocate of open source, is joining forces with this new fork and actively contributing to the project.

RabbitMQ and Apache Kafka are two popular open-source tools that can be used to manage queues, with each having its own strengths and weaknesses depending on the specific use case. RabbitMQ is a good choice for job queues where work is delegated to an asynchronous endpoint, such as resizing an image. It supports multiple protocols, has predefined exchange types, and offers configurable flexible routing. On the other hand, Apache Kafka is more suited for event-driven applications where data must flow between multiple components in the application. It acts as a distributed message bus that allows interested consumers to access and consume messages at any time. Kafka provides great scalability and can handle colossal workloads. The choice between RabbitMQ and Apache Kafka depends on whether the use case involves processing tasks or distributing data throughout an application.