August 2024 Summaries

Postgres 17 is an upcoming release that improves both developer experience and performance. The new MERGE command with RETURNING support allows developers to handle conditional data modifications in a single step, simplifying complex workflows. Enhanced JSON functions enable more straightforward ways to work with semi-structured data, while improved performance features include optimized parallel query processing, vacuum operations, and B-tree index performance. Contributions from Neon engineers also enhanced the Postgres 17 release, including new options for performance tuning, support for parallel builds of BRIN indexes, and improvements in libpq/psql. The upcoming release is planned to be supported by Neon as soon as it's GA.

Postgres 17 introduces several enhancements aimed at improving both developer experience and performance, with key features including the MERGE command with RETURNING support, enhanced JSON functions, and improved overall performance. The MERGE command evolution simplifies conditional data modifications by allowing developers to retrieve modified rows in a single query, while new JSON functions facilitate easier manipulation and querying of semi-structured data. Performance improvements are evident in faster query execution, especially through parallel processing, vacuum operations, and B-tree index handling. Contributions from Neon engineers have resulted in additional features such as the EXPLAIN (SERIALIZE) option for detailed query analysis, parallel BRIN index builds, and enhancements in SSL/TLS connections. These updates, along with ongoing community contributions, reflect Postgres' commitment to evolving as a leading database solution.

The evolution of multi-tenant data architectures in databases like Postgres has led to three main strategies: shared schema, schema-per-user, and database-per-user, each with its advantages and challenges. The database-per-user approach, while historically expensive and complex, offers complete data isolation and is becoming more feasible with advancements in cloud technologies such as Neon, which provides serverless architecture and instantly deployable projects. This approach is favored for its simplicity in security models and compliance with geographic data restrictions, albeit with higher operational costs. The series aims to explore how Neon addresses some of the traditional challenges of database-per-user designs, including the management of user databases, application environments, and the implementation of control planes to automate and streamline processes like onboarding, monitoring, and maintenance. While isolated environments provide maximum flexibility especially in dealing with issues like the noisy neighbor problem, shared environments simplify operations but may compromise on certain capabilities, highlighting a trade-off between operational complexity and flexibility.

The exploration of multi-tenant architectures in Postgres and Neon outlines three primary approaches: shared schema, schema-per-user, and database-per-user. Each has its pros and cons, with database-per-user offering complete data isolation, making it appealing for industries requiring stringent data privacy, despite its historically high costs and complexity. Neon’s serverless design and features address some challenges of database-per-user architectures, offering a more scalable and manageable solution. Isolated application environments provide flexibility but are costly, while shared environments simplify operations but risk compromising data isolation. The series will delve into control planes essential for scaling multi-tenant applications effectively, emphasizing Neon’s role in enhancing these systems.

Both Aurora Serverless v2 and Neon are serverless Postgres databases that offer dynamic scaling of compute resources based on demand. However, there are some key differences between the two platforms. Aurora Serverless v2 adjusts compute capacity in real-time using units called Aurora Capacity Units (ACUs), with a maximum capacity of 128 ACUs and minimum capacity of 0.5 ACU. It scales up and down in increments of 0.5 ACU, and can handle scaling nondisruptively without interrupting ongoing connections or transactions. Neon, on the other hand, measures compute capacity in Compute Units (CUs), with a maximum capacity of 10 CUs and minimum capacity of zero. It scales up and down in increments of 0.25 CU, and also offers nondisruptive scaling during capacity adjustments. Neon's pricing is subscription-based, while Aurora Serverless v2 uses a pay-per-use model based on ACU-hours consumed. Overall, Neon's more responsive scaling algorithm, ability to scale down to zero, and lower compute unit price make it a more cost-efficient option for running serverless Postgres databases.

Neon, a serverless database powered by Postgres, is transitioning its partnership with Vercel to provide all Neon features via the Vercel Marketplace. This change will give Vercel developers access to advanced features such as database branching, autoscaling, and improved security configurations while maintaining unified billing. The migration process will be executed in two steps: auto-migration of existing databases and allowing new databases to be created via Neon in the Vercel Marketplace. This collaboration aims to simplify platforms for developers as they build the next generation of applications.

Neon is enhancing its partnership with Vercel by transitioning Vercel Postgres users to the Neon platform via the Vercel Marketplace, allowing developers access to Neon's comprehensive features such as database branching, serverless compute configurations, and advanced security measures. This move aims to provide a streamlined, integrated frontend cloud experience without disrupting existing infrastructure, as the migration will occur with zero downtime and unified billing. Starting in November, Vercel will begin a phased migration of user accounts to Neon, ensuring users can access new features seamlessly. This initiative also addresses growing interest from Enterprise customers, with plans to simplify the service adoption and procurement processes through the Vercel Marketplace.

Neon's autoscaling feature dynamically extends memory for HNSW index build operations, improving performance while reducing costs. This is particularly useful for applications that require vector similarity searches and Retrieval-Augmented Generation (RAG) apps. By allowing Postgres instances to scale up during high memory and CPU demands, Neon ensures optimal performance without constant overprovisioning. Autoscaling is available in all pricing plans and can be used with the free account.

Neon's autoscaling feature for Postgres databases optimizes performance and cost-efficiency by dynamically adjusting resources based on demand, particularly useful for handling the high memory and CPU requirements of Hierarchical Navigable Small World (HNSW) index builds. This autoscaling ability mitigates the need for constant overprovisioning by using disk swaps to extend memory when necessary, making it possible to efficiently manage large index builds even with limited resources. By scaling up during resource-intensive operations like vector similarity searches and scaling down during regular operations, Neon ensures a cost-effective and efficient user experience. The feature is available across all pricing plans, including the free version, and is particularly beneficial for applications that rely on vector similarity searches for tasks like Retrieval-Augmented Generation (RAG) in large language models.

Neon Autoscaling is now generally available for all Neon databases, including the free plan. This feature automatically adjusts CPU and memory based on workload, ensuring optimal performance and cost efficiency. It eliminates the need for manual server resizing and prevents budget waste from overprovisioning. The autoscaling algorithm is deeply integrated into Neon's architecture, which natively decouples storage and compute. This feature enables Neon databases to autoscale in milliseconds without requiring users to wait for VMs to scale up or down.

Neon has launched its autoscaling feature for Postgres databases, moving out of beta and becoming generally available, including on the Free plan. This development allows Neon Postgres databases to automatically adjust CPU and memory resources in response to workload demands, ensuring optimal performance and cost efficiency while eliminating the need for manual server resizing. The autoscaling feature is deeply integrated into Neon's architecture, utilizing VMs within a Kubernetes cluster to provide strong isolation, dynamic resource adjustments, and seamless live migration. The autoscaler-agent plays a key role by monitoring metrics every five seconds to inform scaling decisions, ensuring efficient resource allocation. Neon's autoscaling addresses common challenges faced by managed Postgres users, such as overprovisioning and manual resizing, by enabling dynamic scaling that aligns with actual usage, thereby reducing costs and improving performance. Users can define maximum and minimum autoscaling limits to balance costs and performance, with the flexibility to scale down to zero for non-production databases. This advancement reflects Neon's commitment to providing a more efficient and cost-effective Postgres solution that adapts to varying workloads without the need for manual intervention.

Rubric Labs is an applied AI lab that focuses on developing AI-augmented solutions for companies. They use Neon, a serverless Postgres database, to streamline their development workflow and enhance the developer experience. Rubric Labs' internal projects include the Rubric Operating System (ROS), which manages operations such as lead management and client relationships. For client projects, they have built SyncLinear, an open-source project that syncs tickets between Linear and GitHub. The company prefers using modern devtools like Next.js for full-stack development, TypeScript for language, Vercel for hosting, Tailwind CSS for styling, and various AI tools such as AI SDK by Vercel, LangChain, Anthropic's Claude model, OpenAI's GPT-4, and pgvector for vector database capabilities within Postgres.

Rubric Labs is an applied AI lab that specializes in developing AI-augmented solutions for companies by rapidly creating experimental proof-of-concept software and full-stack applications, with a particular emphasis on a developer-first experience. They favor modern, open-source development tools and have adopted Neon, a serverless Postgres database, for both internal and client projects due to its features like database branching, autoscaling, and a streamlined development workflow. This allows each developer to work in isolated environments, facilitating parallel testing and quick rollbacks. In addition to Neon, Rubric Labs employs a technology stack that includes Next.js for web development, TypeScript, Vercel for hosting, and Tailwind CSS for styling. They enhance their AI capabilities with tools like Vercel's AI SDK, LangChain, and models from Anthropic and OpenAI. Rubric Labs has also built internal tools like the Rubric Operating System and open-source projects such as SyncLinear, which syncs tickets between Linear and GitHub. They have a strong preference for tools that prioritize developer efficiency and offer solutions like Supabase for rapid iterations and PlanetScale for MySQL projects.

The incident review discusses a recent outage in the us-east-1 region of Neon's services, which resulted in up to 2 hours of unavailability for approximately 0.4% of customer projects. The outage was caused by an EC2 instance failure, and it took around 30 minutes between the initial node failure and the decision to migrate projects away. The incident highlighted the need for a more resilient system, particularly in terms of fault tolerance and response time. To address this, Neon is introducing a new service called the Storage Controller, which uses a reconciliation-loop model to schedule users' Projects onto pageservers and can respond dynamically to changes in node availability or load. The Storage Controller has been in production since May 2024 and is being used to manage high-capacity Projects first, with plans to migrate all paying customers to it in the near future.

On August 14, 2024, an outage affected approximately 0.4% of customer projects in the us-east-1 region for up to two hours due to the failure of an EC2 instance hosting a Neon pageserver. Although the incident was resolved by migrating projects from the failed pageserver to others, the response time was longer than desired, with some customers experiencing up to a two-hour service gap. The delay was attributed to the latency in alert systems and the semi-manual process of migrating projects. Neon has since implemented a new Storage Controller designed to improve fault tolerance and reduce downtime by autonomously managing project migrations in the event of server failures. This new system, which has been in use since May 2024, enables rapid and autonomous responses to node failures, minimizing service disruptions. The company is accelerating the transition of all customer projects to this new infrastructure to enhance resilience and support critical workloads.

Wordware is an AI development platform that streamlines prompt engineering and allows users to build AI applications quickly by writing prompts in plain English. It offers a user-friendly interface, similar to Notion, for collaborative AI development. The platform also provides useful templates to help users get started. Wordware uses Neon, a serverless Postgres database, which enables developers to create previews on real data with minimal setup and identify issues early in the development process. This approach ensures stability in production while allowing rapid iteration and deployment of AI applications.

Wordware, a platform that enables rapid development and deployment of AI agents, streamlines the process of creating AI applications by offering a user-friendly interface similar to Notion. By focusing on natural language programming, Wordware allows users to write prompts in plain English and collaborate with domain experts, such as salespeople or lawyers, to refine their AI agents. The platform leverages Neon, a serverless Postgres database, to enhance development speed and efficiency by using features like scale-to-zero, autoscaling, and database branching for real-time previews with real data. This setup allows Wordware to catch and fix issues early, reducing the need for emergency fixes and ensuring stability in production while maintaining the flexibility of traditional Postgres. Co-founders Robert Chandler and Filip, inspired by their experiences with AI and the challenges of implementation, created Wordware to simplify the process of crafting high-quality prompts and to involve domain experts directly in the development process, ultimately making AI development faster and more effective.

In this article, we explored a workflow to test changes in a dev environment in Neon and push the changes to prod in RDS using database branching with Neon. We introduced the concept of database branching, which enables you to instantly create isolated copies of a particular dataset via copy-on-write, allowing you to safely experiment with changes without affecting the main development branch. The article also covered how to document changes made to your dev branches in Neon using schema diff feature and how to automate this process through GitHub Actions. Two methodologies were presented for migrating changes from your Neon Twin back to your AWS RDS production database: Using SQL Files and psql, and Using Prisma ORM. Both methods utilize GitHub Actions to trigger the migration process when a pull request is merged into main. The article concluded with setting up syncing production with Neon Twin using a similar approach as before but rather than on a schedule it will run after production migrations have been applied.

The article explores a workflow for migrating changes from a development environment, Neon Twin, back to a production database on AWS RDS, emphasizing two methodologies: SQL scripts and Prisma ORM. It elaborates on setting up a Neon Twin as a development database and introduces database branching in Neon, which allows developers to experiment safely with isolated copies of production data. The process involves documenting schema changes using Neon's schema diff feature and implementing GitHub Actions to automate the migration either by manually applying SQL scripts with psql or using Prisma's migration tools. Both methods aim to streamline database schema management and improve development efficiency. Additionally, the article discusses re-synchronizing the production database with Neon Twin after migrations to ensure both environments remain aligned.

The psql -h pg.neon.tech command is a feature in Neon that allows users to connect to their projects without needing passwords. When run, it generates a URL that takes the user to a UI where they can choose which project or endpoint to connect to. This command is useful for developers working with multiple Neon projects or endpoints, as it simplifies the connection process and eliminates the need to remember URLs or credentials. The "Link Proxy" service running in the data plane generates this URL and initiates a connection to the correct IP with appropriate credentials. Both the regular Neon Proxy and the "Link Proxy" are open source, allowing users to explore their codebase and ask questions about their architecture on Neon's Discord server.

The "psql -h pg.neon.tech" command is a feature of Neon that enables users to connect to Neon projects seamlessly through a passwordless authentication process. When executed in the command line interface, it generates a URL that directs users to the Neon console, where they can select the desired project or compute endpoint. This functionality relies on a server known as "Link Proxy," which mimics a Postgres instance and uses the Postgres protocol to display a welcome message and facilitate connections. Neon employs a control plane and data plane architecture, where the Link Proxy operates within the data plane to handle connections and forward traffic between the client and the database. This tool simplifies managing multiple projects or endpoints by eliminating the need to track numerous URLs or credentials. Additionally, both the Neon Proxy and Link Proxy are open-source, allowing users to explore the codebase and engage with the Neon community for support.

The text discusses the adaptability and dominance of Relational Database Management Systems (RDBMS) over other data models such as document storage, vector databases, and graph databases. It highlights how RDBMS has incorporated features from these alternative systems, making them more versatile and efficient. The text also emphasizes the simplicity, performance, and community support that contribute to the success of RDBMSes like Postgres. Furthermore, it suggests that developers can use Postgres as a single "data management framework" for various applications by leveraging its adaptability and numerous extensions.

The text explores the enduring dominance of relational database management systems (RDBMS), particularly PostgreSQL, in the face of emerging database technologies like document stores, vector databases, and graph databases. Despite the periodic rise of new data models, such as MapReduce, key-value stores, and document databases, RDBMSs have consistently integrated the novel features of these systems, maintaining their relevance and superiority. The adaptability of the relational model, exemplified by PostgreSQL's support for JSON, graph queries, and vector embeddings, is highlighted as a key factor in its sustained success. The piece also underscores the importance of community contributions to PostgreSQL's development and its ability to handle complex data management needs across various domains. Additionally, the text argues that while alternative database models may offer specific advantages, RDBMSs often outperform them in simplicity and performance due to advances in query optimization and indexing. The overarching message is that PostgreSQL's comprehensive capabilities and strong community support make it a versatile choice for diverse application needs, embodying the concept of "Postgres is Enough."

Logical replication allows you to replicate data from your Neon database to various external systems, enabling Change Data Capture (CDC) and real-time analytics. It's a publish-subscribe model where all changes are tracked via Postgres' Write-Ahead Log (WAL), ensuring that the data in both sources remains synchronized. Logical replication is useful for selective data sync, real-time data replication, production migrations without downtime, and safer Postgres upgrades in large databases. It differs from pg_dump/restore, which creates static snapshots of the database or selected tables, but not ongoing changes. When to use logical replication depends on your needs, as it requires both data sources to be completely in sync in some cases. To set up logical replication in Neon, navigate to project settings, select Logical Replication, and click Enable, verifying that the right roles have the required privilege. Publications define which tables and changes are replicated from the publisher to subscribers, while replication slots ensure data consistency and prevent potential data loss.

Neon has introduced outbound logical replication in beta, enabling users to stream data from Neon databases to various external platforms like Airbyte, Fivetran, Kafka, and others for real-time analytics and Change Data Capture (CDC). Logical replication in Postgres allows data changes, such as inserts, updates, and deletes, to be mirrored in real-time to another database or data source, keeping them synchronized. Unlike pg_dump/restore, which creates static snapshots, logical replication continuously streams changes to ensure data consistency and real-time updates. It supports selective data synchronization, real-time replication, and production migrations without downtime, though it does not handle schema or configuration changes. Setting up logical replication involves creating publications that define which tables and changes are replicated, and managing replication slots to ensure data consistency. Neon handles replication using Postgres' Write-Ahead Log (WAL) and decoder plugins to process data changes. Users can access step-by-step guides to replicate data from Neon to several platforms, and interested users can join the Early Access list for inbound replication.

As the complexity of databases grows, Entity-Relationship Diagrams (ERDs) provide a visual representation of tables, attributes, and relationships between them, making it easier to understand and manage data integrity and referential integrity in Postgres. ERDs can help turn abstract database structures into visual representations, make schema design accessible, and understand the current state of the database. With tools like Neon and Outerbase's automatic diagram generation, databases are becoming more accessible to everyone, including non-technical team members or beginners, allowing developers to better control their data structures, facilitate communication between technical and non-technical stakeholders, and make systems more transparent and robust.

Entity-Relationship Diagrams (ERDs) are valuable tools for visualizing the relationships within complex databases, particularly in PostgreSQL environments, by illustrating entities, attributes, and their interconnections. They simplify understanding by transforming abstract database structures into accessible visual representations, which is especially beneficial for non-technical team members. ERDs emphasize the importance of primary and foreign keys, aiding in designing robust, scalable schemas and enhancing communication among team members. Outerbase's new feature automatically generates and updates ERDs for Neon databases, streamlining the process and making it easier for developers to manage and understand their data structures. This automation supports the democratization of data management, allowing developers to interact more effectively with databases and enabling collaboration with AI to optimize schemas. As databases become increasingly accessible, tools like Outerbase facilitate better data structure comprehension and management, encouraging developers to leverage ERDs for improved data handling and communication.

Neon is coming to Azure with a $25M strategic investment from M12, Microsoft's Venture Fund, as part of its multi-cloud strategy. Neon aims to help developers ship faster with Postgres on the platform, regardless of the cloud provider they use. With this expansion, Neon becomes a leading Postgres platform and emphasizes Microsoft's commitment to integrating it into Azure. The move aligns Neon's focus on delivering a better developer experience with Microsoft's strategy for dev tools, including GitHub, VS Code, and TypeScript. Neon is positioned to support modern developers who want to use Postgres, offering serverless capabilities that remove operational overhead and accelerate development velocity. As the next generation of AI apps rely on vector databases, Neon becomes an ideal choice with its unique serverless capabilities and cost-efficiency. Developers can join the waitlist to be among the first to experience Neon on Azure, or explore open engineering positions to help shape the future of AI and Postgres.

Neon has expanded its database platform by integrating with Microsoft Azure, supported by a $25 million investment from M12, Microsoft's Venture Fund. As a leading Postgres platform, Neon aims to enhance the developer experience by providing a serverless architecture that separates compute and storage, enabling faster development and reduced operational overhead. This integration aligns with Microsoft's strategy of investing in modern development tools and positions Neon as a key player for AI applications through its vector database capabilities, which offer performance, cost-efficiency, and data privacy solutions. Neon is now available as an Azure Native Integration, allowing users to deploy directly from the Azure Portal, marking a significant step in Neon's multi-cloud vision and commitment to supporting developers across different cloud environments.

Genezio has partnered with Neon to offer a managed Postgres database to their developers, enhancing the full-stack app development experience on their platform. This integration provides a seamless and efficient way for developers to deploy and manage applications, while also offering cost-efficiency through usage-based billing and serverless scalability. With Neon's API-first approach and fast deployment times, Genezio can now provide a single-command full-stack deployment experience, making it easier for developers to launch and manage their applications. The partnership aims to improve the developer experience, allowing users to focus on building their application rather than managing infrastructure, while also expanding revenue opportunities through a one-stop shop platform.

Genezio, a function-as-a-service platform focused on serverless architecture, has partnered with Neon to integrate a managed, serverless Postgres database into its offerings, enhancing its full-stack application development capabilities. This integration aligns with Genezio's mission to simplify and optimize the deployment of full-stack applications by providing developers with a seamless, serverless experience that reduces costs and improves responsiveness. Neon's API-first approach and serverless capabilities, such as scale-to-zero and automatic scaling, complement Genezio's needs, offering a cost-efficient, fast, and reliable database solution that supports both hobbyist and commercial projects. Additionally, Neon's partnership program and usage-based billing model facilitate the incorporation of Postgres into developer platforms, allowing Genezio to offer a comprehensive, end-to-end development experience without the need for external database services.

With Neon, Comigo.ai is transforming ADHD support by leveraging AI and serverless technology to build a digital companion and coach that offers personalized productivity and therapeutic support, reducing cognitive load and improving 24/7 accessibility for individuals with ADHD. The company has shifted its focus from database management to app development, thanks to Neon's autoscaling resources, developer-friendly nature, and branching capabilities, which enable efficient management of development environments, improved CI/CD processes, and bug reproduction. Comigo.ai is now able to build a more effective companion for people with ADHD, such as themselves, by focusing on directly achieving product market fit, rather than managing operational overhead.

Comigo.ai, a digital companion app designed to support individuals with ADHD, utilizes Neon's serverless Postgres platform to enhance its functionality and scalability. The app provides AI-powered body doubling, therapeutic exercises like CBT and DBT, and personalized task management, aiming to reduce cognitive load and improve productivity for users with ADHD. Initially facing challenges with Cloud SQL for PostgreSQL due to operational overhead, Comigo transitioned to Neon, benefiting from its automatic scaling and developer-friendly features. Neon’s database branching capability allows Comigo to efficiently manage development environments, improve CI/CD processes, and safely reproduce bugs without impacting the production environment. This transition has enabled Comigo to focus more on achieving product-market fit rather than managing database infrastructure.

This article guides you through setting up Slack notifications to monitor pg_dump and restore workflows. By automating nightly dump/restores from RDS to Neon, real-time job status alerts can be sent via Slack. The process involves creating a Slack channel, setting up and configuring a Slack App using the Slack developer console, modifying your existing GitHub Actions workflow, developing JavaScript functions to post formatted notifications to your Slack channel, and committing changes and deploying the action. This setup allows for real-time updates on pg_dump and restore activities, improving communication and efficiency within teams.

The article outlines a detailed process for setting up Slack notifications to monitor pg_dump and restore workflows from an RDS production database to a Neon environment, using GitHub Actions. By creating a Slack Webhook, teams can receive real-time alerts in a Slack channel about the status of these database operations, avoiding the need for GitHub email notifications. The guide includes instructions on setting up a Slack app, configuring GitHub Actions with JavaScript functions for success and failure notifications, and using environment variables to access job details like database size, job duration, and names. Additionally, it provides steps for deploying the action on GitHub, including updating repository secrets with the Slack Webhook URL. The article also mentions the broader context of using Neon for development, testing, and staging, while keeping production databases on RDS, and is part of a series on optimizing AWS RDS environments.