November 2022 Summaries

Huuva, a startup that provides local restaurants with an outlet for their products through its kitchens, uses Apache Kafka® to streamline order management and improve efficiency in the kitchen. The company's software helps chefs queue up orders so that the kitchen can operate smoothly and efficiently without hurrying. Aiven's Cluster program has been instrumental in helping Huuva set up their services in a sustainable, scalable way, allowing them to focus on providing great features for their kitchens as they continue to grow.

Apache Kafka Connect is a powerful tool that enables integration of Apache Kafka with various technologies as data sources or sinks by defining configurations in JSON files. However, its complex configuration and partially overlapping functionality can make it seem like dark magic to new users. To become proficient in using Kafka Connect, one must read the manual thoroughly and understand the different connectors available for a specific integration problem. Here are some tips to improve your developer experience with Apache Kafka Connect: 1. Prepare the data landing ground by pre-creating all necessary data structures before integrating them with Apache Kafka. Avoid using auto_create_topics_enable or auto.create features, as they may lead to loss of control over these artifacts and generate problems in downstream pipelines. 2. Evaluate the benefits, limits, and risks of various connectors to choose the best one for your needs. For example, when sourcing database data into Apache Kafka, consider using either a polling mechanism based on JDBC queries or Debezium's push mechanism. 3. Check all pre-requisites before starting the connector, including network paths, credentials and privileges, and required objects in place. Ensure that all necessary JAR dependencies are placed in the correct folder. 4. Use data formats that specify schemas to avoid errors when sinking data to technologies requiring a schema. Tools like Karapace can be used for this purpose. 5. Utilize Single Message Transformations (SMT) to reshape the data payload during integration, allowing filtering, routing, defining keys, and masking of data. 6. Define keys properly to drive data partitioning and lookups in both source and sink environments. This can help achieve better performance and correctness. 7. Increase the connector's robustness by reducing the amount of data in flight, parallelizing the load, and knowing how to debug errors effectively. 8. Keep an evolution trace of your configuration changes using version control systems and automate deployment as much as possible. By following these tips and exploring additional resources like Aiven for Apache Kafka Connect and How To Guides for Source and Sink Connectors, you can master the art of integrating data sources and sinks with Apache Kafka using Kafka Connect.

Aiven, an AWS partner providing fully managed open source services like Apache Kafka®, OpenSearch®, and Cassandra®, is participating in Amazon Web Service's (AWS) annual event re:Invent. The event will take place in Las Vegas and focus on cloud technology, innovation, and information sharing. Aiven will be present at Booth 2347 to discuss its services and integration with AWS offerings like CloudWatch, S3, and Private Link. Additionally, Aiven's CISO Jamie Arlen will give a talk on "Modern AppDev and AppSec cloud solution architecture strategy" during re:Invent.

Fast-scaling companies face specific challenges such as attracting and retaining developer talent, ensuring high development productivity, building scalable processes and workflows, and addressing cybersecurity threats. Tech leaders must address these growing pains to maintain the company's growth. Solutions include focusing on developer experience, automating mundane chores, investing in self-service infrastructure tooling, adopting product thinking for internal technical platforms, setting aside time to review processes and workflows, planning ahead for future tech needs, and ensuring data security through a security-by-design approach. Managed data platforms like Aiven can help ease these growing pains by outsourcing some of the vital steps involved in addressing these challenges.

Aiven's Open Source Program Office (OSPO) is dedicated to maintaining and contributing to open source projects that impact the company. The OSPO team actively collaborates with external contributors, promotes open source processes and philosophy within Aiven, and encourages open-source practices. They currently contribute daily to Apache Kafka®, Apache Flink®, PostgreSQL®, Apache Cassandra®, and OpenSearch® communities. Additionally, the team maintains several Aiven-owned open-source projects such as Klaw, Guardian for Apache Kafka®, PGTracer for PostgreSQL®, and Prometheus® exporter plugin for OpenSearch®. The OSPO is divided into chapters focusing on each open source project or ecosystem, with developers sharing their expertise across different topics.

Aiven, a cloud-based data infrastructure service provider, ensures that customers get the best possible price/performance ratio by constantly fine-tuning the performance of all services throughout their lifecycle. As hardware continues to evolve rapidly, cloud platform providers introduce new virtual machine instance types optimized for different use cases. Aiven takes the burden away from its customers by tracking this evolution and evaluating which instance types offer the optimal price/performance ratio for specific services. When a better instance type is found, it is brought into production, and existing services switch to the new instance type during maintenance updates. This process has recently led to improved Redis performance on Aiven's platform.

This blog uses an ice-cream store metaphor to explain different data management techniques such as streaming, batch processing, caching, archiving, and encryption. It compares fresh milk with fresh data, powdered milk with cached data, and the concept of a constant supply of fresh milk with stream processing. The analogy helps readers understand how these techniques can be used in their enterprise data architecture. Additionally, it discusses long-term storage solutions for archival data and handling sensitive data securely.

Aiven has published its first sustainability update after committing to being a sustainable data infrastructure provider. The company is focusing on understanding and measuring its carbon footprint, building diversity, equity, inclusion, and belonging within the organization, and driving change through social impact initiatives. Aiven's Social Impact initiative supports external projects that aim to improve society and utilize open source technologies. The company has also introduced a core sustainability team and plans to present a detailed report on its findings in early 2023.

Aiven, a cloud technology company, has partnered with Microsoft to provide its services directly through the Azure Marketplace. This allows customers familiar with the Azure ecosystem to easily access and leverage their long-term commitment to Microsoft Azure for Aiven services on the Azure platform. By signing up via the marketplace, users get the same level of access to Aiven control plane and services as when signing up directly through Aiven's website. Payments are automatically managed through the Azure billing account, with prices being the same as direct sign-up. The subscription does not include any monthly fees and is charged hourly. To get started, users need to subscribe using their Azure AD user account and then proceed to set up the subscription on the Aiven Console.

Aiven has launched a Sustainability and Social Impact program aimed at combating climate change and promoting equal access to technology. The company's goals include providing clean and efficient data infrastructure, promoting diversity and inclusivity, lowering barriers to new technologies, and fostering healthy communities around open source projects. To achieve these objectives, Aiven has created a dedicated team of professionals leading specific tracks within the program. The company is also working on measuring its carbon footprint, improving internal travel policies, and supporting social impact initiatives through engagement with open-source projects and nonprofit organizations.

Aiven, a company focused on open source and managing it efficiently, has contributed back to the Apache Kafka® community with a dedicated full-time team. The team includes Matthew de Detrich, Chris Egerton, and Greg Harris who work on open source projects. Aiven's Open Source Program Office (OSPO) is committed to working on open source sustainability by fostering communities, employing developers for open source projects, and compensating employees for contributing to such projects even if it doesn't align with their role at Aiven. Chris Egerton, a member of the Apache Kafka® team, has been invited as a committer by the Project Management Committee for Apache Kafka. His notable contributions include KIP-507 and KIP-618, which focus on securing internal connect REST endpoints and exactly-once support for source connectors respectively. Aiven's commitment to open source is evident through its work with various projects such as PGhoard, PGlookout, Karapace, Klaw, and more.

An API is crucial when making cloud purchasing decisions, as it ensures compatibility and future-proofing of the chosen services. Cloud-native applications should compose their platforms from the best components available, considering not only current needs but also future growth and evolution. Choosing cloud services with APIs allows for seamless integration between different systems and freedom to build custom integrations. This is similar to buying devices with standard ports and connectors, ensuring compatibility when adding new components in the future.

An API (Application Programming Interface) is crucial when considering cloud platform purchases. Cloud-native applications should be composed of the best components available to meet their needs. When making decisions about how to compose a solution, it's important to consider not only current requirements but also future growth and evolution. Choosing cloud services with APIs allows for seamless integration between different platforms and ensures that new components can be added without compatibility issues. This is similar to buying devices with standard ports and connectors, allowing users to replace or add components without needing to change the entire setup.

Aiven has recently measured the throughput performance of Apache Kafka on its platform, updating benchmarking results from 2017 and 2019. The tests were conducted on three standard plans - Business-4, Business-8, and Premium-6x-8 - across three major cloud providers: AWS, GCP, and Azure. The goal was to determine the best plan for various application workloads. The benchmarking results showed that the maximum sustainable write throughput varied depending on the cloud provider and the Aiven plan used. For example, in the Business-4 plan, the maximum bytes in rate from clients were 37.0 MB/s (AWS), 30.5 MB/s (GCP), and 32.7 MB/s (Azure). In contrast, for the Premium-6x-8 plan, the respective figures increased to 123 MB/s, 147 MB/s, and 137 MB/s. The cost per MB/s per month remained relatively consistent across different plans, with only a slight increase as the plan's capacity was upgraded. Aiven for Apache Kafka is designed to ensure that it does not hit IOPS limits in any 24-hour period, making it suitable for production workloads.

James Job AG is a startup that uses Apache Kafka® to revolutionize job searching by matching job openings to users' skill sets. The company faces challenges in managing the high volume of data generated by its architecture, which calculates many things behind the scenes and triggers numerous calculations when changes occur. Event streaming with Apache Kafka® helps address these challenges by providing a solid infrastructure for handling large amounts of data, enabling rapid development and flexibility, and offering infinite retention capabilities. The event-driven approach also allows James Job to minimize user impact during system downtime, replay historical events for new use cases, and build more resilient, stable, and flexible applications.

This article provides a detailed tutorial on how to use PostgreSQL JSON functions to analyze Indian restaurant reviews data. It covers the basics of setting up a PostgreSQL instance, loading the dataset into the database, and using various SQL queries with jsonb_array_elements and other functions to extract and filter information from the nested JSON documents. The tutorial also demonstrates how to create tables based on the parsed data and perform more complex analysis such as creating histograms and filtering by specific criteria like price range and user ratings.

Event-driven programming is a paradigm where entities communicate indirectly by sending messages to one another through an intermediary, typically stored in a queue before being handled by the consumers. This approach completely decouples the producer from the consumer and allows for multiple producers and consumers to collaborate on processing incoming requests. It also simplifies retries and maintaining event history while making it easier to scale large systems. Key components of an event-driven system include events, producers, consumers, and message queues. Event-based programming opens the door to several useful patterns such as single producer/single consumer, single producer/multiple consumers, dead letter queue (DLQ), time to live (TTL), and asynchronous request-response with events.

On October 25, 2022, a potential critical OpenSSL vulnerability was discovered, with official details published on November 1, 2022. Two high-severity vulnerabilities (CVE-2022-3786 and CVE-2022-3602) affecting OpenSSL v3.0-3.6 were identified, which could lead to buffer overruns resulting in denial of service or remote code execution. Exploitation requires a malicious certificate signed by a certificate authority or an application continuing verification despite failure to construct a path to a trusted issuer. Aiven services and the platform have been thoroughly investigated for potential vulnerabilities, but no impact has been found.

On October 25, 2022, a potential critical OpenSSL vulnerability was discovered, affecting OpenSSL v3.0-3.6. The vulnerabilities could lead to buffer overruns resulting in denial of service or remote code execution. Aiven's security, engineering, and operations teams have investigated the potential impact on their platform and services and found no vulnerable components. CVE-2022-3786 and CVE-2022-3602 currently have no impact on Aiven services or the Aiven platform.

The 12th annual PostgreSQL Conference in Europe took place on October 25-28 in Berlin, Germany. Aiven's team attended the event to meet with the Postgres community and participate in discussions around PostgreSQL use cases and latest developments. Key highlights from the conference included keynotes on efficient graph analysis with SQL/PGQ, improvements to BRIN indexes, PostgreSQL use cases at GitLab, and various technical presentations on topics such as performance optimization, failover of logical replication slots, and full page writes in PostgreSQL. The event also saw an increase in the number of active users, participants in public forums, and contributors to the PostgreSQL community.

This article discusses the importance of automating data infrastructure and highlights five reasons to do so: reliability for data infrastructure, predictable automation tool, consistency across environments, security from automation, and cost and agility. It also introduces Terraform as an open-source IaC tool that can help automate the provisioning of application and data infrastructure in a multi-cloud deployment model. The author suggests starting with a small proof-of-concept to implement these concepts within an organization.