July 2022 Summaries

In this tutorial, learn how to use OpenSearch® to perform both simple and advanced searches on semi-structured recipe data, and from that produce the perfect menu. The author demonstrates how to find delicious recipes in a Pythonic way using a high-quality dataset from Epicurious containing over 20,000 full recipes, ratings, and nutrition information. They use Aiven's fully managed OpenSearch service to get their cluster up and running and provide a demo that contains all the code to connect, send data, and perform search queries. The tutorial covers various search techniques such as range function, match_phrase, slop parameter, and match function with "AND" operator.

In April, Aiven held an internal hackathon during their company off-site event. The day began with a LEGO® icebreaker exercise designed to simulate software development and maintenance through building exercises. Participants were divided into teams of 4-5 people who had to build a cafe or house within specific requirements in 15 minutes. Afterward, teams swapped projects and continued building with modified requirements. The hackathon focused on making new connections and contributing to open source projects. A total of 57 net new PRs were merged, and numerous issues were created, considered, and closed. Some notable public projects that participated include Guardian for Apache Kafka, several Kafka connectors, Karapace, the Metadata Parser, Prometheus exporter plugin for OpenSearch, and Rohmu.

The case for streaming over batch data continues to strengthen as more companies adopt modern data stacks built around streaming and stream processing. Data teams moving towards streaming benefit from improved data timeliness, quality, cost savings, and reduced operational overhead. Streaming pipelines enable real-time applications, enhance data completeness through CDC, and allow for pre-aggregation and enrichment of data in transit, leading to significant storage and compute cost savings. Additionally, stream processing can help maintain data integrity during downtime and ensure a more reliable data interface for business decisions.

Apache Kafka is a widely-used event streaming platform that is distributed, scalable, high-throughput, low-latency, and has a large ecosystem. It serves as a transportation mechanism for messages across multiple systems or microservices. The platform is distributed, meaning it relies on multiple servers with data replicated over multiple locations to ensure resilience against server failures. Apache Kafka is scalable, allowing users to add more servers as their system grows. The technology's community and ecosystem include client libraries for various programming languages and a set of data connectors to integrate Kafka with existing external systems. This makes it easier for developers to start using Apache Kafka without having to reinvent the wheel. Apache Kafka is commonly used in scenarios where real-time events need to be processed, such as e-commerce projects that require immediate recommendations based on users' latest purchases. The platform helps untangle data flows and simplify handling of real-time data while decoupling subsystems. The approach taken by Apache Kafka is to describe entities using continuously arriving events rather than static objects or aggregated facts stored in a database. This allows for replaying events, answering different types of questions about the products and sales, and maintaining an event-driven architecture. Apache Kafka's push-pull model involves producers that create and push messages into the cluster and consumers who pull, read, and process the messages. Producers and consumers can be written in different languages and platforms, allowing for flexibility and decoupling of systems. Data is organized within Apache Kafka using topics, which are abstract terms representing sequences of messages. Topics can have multiple partitions stored across multiple machines to enable horizontal scaling. Replication ensures high availability and prevents data loss by replicating data across brokers at the partition level. Apache Kafka connectors simplify connecting applications to Apache Kafka by integrating external data sources, such as databases and tools, using pre-built connectors or custom ones.

The UK Government has selected Aiven for OpenSearch to run search services on its expansive GOV.UK Platform as a Service (PaaS). Since 2018, Aiven has been working with the Government Digital Service (GDS) to provide open source streaming and data management technologies combined with cloud infrastructure. The GOV.UK PaaS was created by GDS in 2015 and hosts over 60 departments and public sector organizations. A key requirement for the platform is a powerful search service, which led the team to choose Aiven for OpenSearch due to its alignment with open source technologies and ability to meet the GOV.UK PaaS requirements.

In this blog post, James Arlen, Aiven's CISO, discusses his thoughts on Pride Week and what it means for him as a leader in the company. He emphasizes the importance of embracing vulnerability and admitting that there is always more to learn about creating an inclusive workplace for 2SLBGTQIA+ individuals. Arlen shares his experiences with connecting members of Aiven's #ProudCrabs community with the Executive Team, diversifying the company's staff, and ensuring a safe environment free from harassment and inappropriate behavior. He encourages other non-minority leaders to take advantage of learning opportunities and support their 2SLBGTQIA+ colleagues throughout the year.