April 2024 Summaries

As companies scale, their database costs can skyrocket due to increased demand for resources. One often overlooked aspect is compute costs, which typically account for a significant portion of managed Postgres bills. To reduce these costs, it's essential to choose the right instance size and optimize your Postgres database through query optimization and caching. Additionally, using autoscaling and autosuspending features can help match costs with load without manual intervention. Neon offers such capabilities, allowing instances to scale down to zero when not in use and dynamically adjusting compute resources based on demand.

As startups scale and face increasing database costs, particularly with Postgres, compute expenses can become a significant burden despite decreasing storage prices. Neon addresses this challenge by offering autoscaling and scale-to-zero features that adjust compute resources dynamically, ensuring payment only for the resources used without compromising performance or requiring manual intervention. In contrast, Amazon RDS, a popular managed Postgres service, ties compute costs to instance size and usage type, often leading to overprovisioning. To optimize RDS compute costs, strategies include choosing the right database strategy, rightsizing instances, and optimizing queries, though these approaches are time-consuming and error-prone. Neon simplifies this process by using a combination of autoscaling, autosuspend, and a re-architected serverless Postgres solution, allowing for efficient management of isolated databases and reducing unnecessary compute expenses.

Neon has released Serverless Driver 0.9.0 for Deno's JavaScript Registry (JSR). JSR is designed to work with TypeScript, ECMAScript Modules, and new JavaScript runtimes like Cloudflare Workers and Bun. The serverless driver allows V8-isolate runtimes such as Deno and Cloudflare Workers to connect to the database via protocols other than TCP, offering faster query latencies. It can be added to projects using the "deno add" command and is compatible with multiple JavaScript runtimes including Deno, Cloudflare, Bun, and Node.

Neon's SQL Editor now supports Postgres's introspection commands such as \d and others. These commands were originally implemented in the C code for psql client but have been translated into JavaScript to enable their use within web browsers. The implementation includes reimplementing some C and Postgres functions, writing examples of all different psql introspection commands, and a test script that compares their results in psql to the results generated by the new JavaScript code. This feature allows users to list or describe things held in Postgres databases directly from their web browser's SQL Editor.

Neon introduces Time Travel Queries for Postgres, allowing developers to query their database at any time within the configured history retention window. This feature is available in the SQL Editor on console.neon.tech and can be enabled using a toggle. Time Travel Queries enable developers to diagnose and debug complex issues by providing a complete picture of the series of events leading up to the current situation. The upcoming feature will allow specifying a timestamp or LSN in the connection string to connect to ephemeral branches, making it possible to perform point-in-time recovery and use favorite tools for time travel queries.

Neon has released its Serverless Driver 0.9.0 on Deno's JavaScript Registry (JSR), marking the introduction of a fast, Edge-compatible Postgres driver designed to optimize query latencies for V8-isolate runtimes such as Deno and Cloudflare Workers. This driver allows connections to databases via WebSocket instead of the traditional TCP, significantly reducing query latency times to sub-10 milliseconds. The JSR is an npm-like registry optimized for TypeScript and ECMAScript Modules, catering to new JavaScript runtimes like Cloudflare Workers and Bun. The serverless driver can be added to projects using the "deno add" command, and it supports integration across multiple JavaScript environments, including Deno, Bun, and Node. The Neon team encourages developers to try the driver, offering free trials and support through platforms like Discord, while also providing deployment options via Deno Deploy and GitHub integration.

Neon has introduced a Time Travel Queries feature in its SQL Editor, allowing users to query their Postgres databases at any point in the recent past, aiding in diagnosing and debugging complex issues. This feature is part of Neon's broader capabilities, which include creating instantaneous development and preview environments with their branchable Postgres and autoscaling production environments. By enabling Time Travel, users can use a timestamp selector to access their database within a configured history retention window, making it possible to identify and restore lost data. This is achieved through Neon's architecture, which separates storage and compute, allowing for near-instantaneous point-in-time recovery using the Write-Ahead Log (WAL) processed by Safekeepers and Pageservers. Neon plans to enhance this feature by enabling users to connect to ephemeral branches directly through a timestamp or LSN in their connection string, facilitating database dumps and connections to prior database versions, while ephemeral branches help manage compute and storage costs.

Neon's SQL Editor has been upgraded to support Postgres's introspection commands, like the psql backslash commands, through a serverless driver that operates in web environments without raw TCP connections. Initially, these commands, implemented in the psql client C code, were not supported in the web-based SQL Editor, leading to syntax errors. To address this, the C code for these commands was translated into JavaScript using regular expressions, allowing them to function in the browser. This development involved parsing SQL queries in the client-side to handle introspection commands separately and creating an open-source package to efficiently identify comments and statement-terminating semicolons in SQL strings. Users can now utilize these introspection commands in the Neon console, with additional features like viewing underlying SQL queries when running commands. The initiative reflects a pragmatic approach to integrating Postgres features into Neon’s web-based tools, aiming to enhance user experience and efficiency in managing database queries.

Schema Diff` in Neon allows developers to easily track changes to their database schema by comparing schemas between the current and past state of a branch, providing benefits for debugging, code review, and cross-team collaboration. This feature creates short-lived, ephemeral branches that can be used to compare the current and previous states of a branch. Additionally, `Protected Branches` in Neon prevent unauthorized access to sensitive data by restricting database access to trusted IP addresses or other rules, allowing developers to build with confidence without compromising development velocity. These features are designed to provide greater control over database security and schema changes, enabling teams to ship with confidence while maintaining development velocity.

Neon has introduced enhanced monitoring and organizations features for its serverless Postgres service. The new monitoring page in the Neon console allows developers to track their database's performance, including CPU usage, RAM consumption, connections count, buffer cache hit rate, deadlocks, and rows. This helps teams understand their application's baseline performance and general trends, enabling them to ship with greater confidence. Additionally, Neon has added Organizations to the console in Private Preview, allowing customers on Launch and Scale plans to request access by contacting [email protected]. The new feature enables users to create a collection of projects within an organization, facilitating collaboration among team members.

Neon has announced new features to enhance the experience of using its serverless Postgres, including a monitoring page on the Neon console and an Organizations feature, currently in Private Preview. The monitoring dashboard offers insights into various metrics such as CPU, RAM, connections count, and database size, helping developers understand application performance and identify potential issues. The Organizations feature enables collaborative project management by allowing users to share projects within a collection, enhancing teamwork. Users interested in early access to Organizations can contact Neon’s customer success team. These updates aim to improve developer velocity and facilitate efficient application management on Neon.

Neon introduces two new features, Schema Diff and Protected Branches, which enhance development workflows and data security within its database branching environment. Schema Diff allows users to compare database schemas over time, similar to Git diffs, aiding debugging, code review, and collaboration by visually displaying schema changes. This feature utilizes Neon's storage engine to create short-lived branches that enable time travel queries for effective schema comparison. Protected Branches, available on the Scale plan, restrict access to sensitive data by allowing only trusted IP addresses, with plans to introduce additional protection rules. Together, these features enable developers to manage schema changes and secure sensitive data more confidently, enhancing the ability to develop and ship projects efficiently using Neon.

Neon simplifies the use of scalable Postgres by changing how developers handle their database infrastructure from development to production. This allows for increased developer velocity and focus on building value for customers, while maintaining Postgres compatibility with features like database branching, autoscaling, and serverless scaling to zero. Neon minimizes Total Cost of Ownership (TCO) in Database Infrastructure by eliminating the need for complex architecture involving virtual private clouds, subnet configurations, and manual replication configurations across availability zones. It also enables true serverless architecture that can scale to zero, increasing cost savings and reducing complexity. With Neon's database branching, developers can clone databases in seconds, making it possible to perform operations on branches independently without impacting the original data. This empowers developers to take a more iterative and experimental development approach, allowing for rapid prototyping and testing without risk to the production environment. Neon integrates well with developer workflows, reducing friction associated with database-related operations and enhancing developer velocity through its command line interface and API. Additionally, it enables connection pooling, autoscaling, and serverless scaling, making it an instrumental tool to enhance the scalability and functionality of applications and workloads.

We implemented storage sharding in our open-source Postgres multi-tenant storage engine, Pageserver, to support large datasets without competing resources on the same server. This allows us to distribute data across multiple Pageservers, delivering higher storage capacity and throughput, while maintaining invisible configuration for developers. Storage sharding has several advantages, including improved performance for smaller databases, reduced impact of maintenance operations, and increased scalability for larger datasets. The implementation involves dividing data into shards based on a pseudorandom hash function and storing them on separate Pageservers, with the ability to split shards and optimize read and write paths through shard subscription and routing. Our current architecture allows us to scale tenants to tens of terabytes of data and potentially hundreds of terabytes in the future, but we plan to implement further optimizations such as WAL fan-out and shard splitting to further improve performance.

Neon is now Generally Available, offering improvements to Postgres internals and addressing scalability issues. It aims to provide a better developer experience for building database applications, prioritizing efficiency and engineering velocity. Neon's architecture separates storage and compute, allowing for faster scaling, branching, and restoration of databases. This enables developers to work on different branches without conflicts, and scales independently of storage, reducing the need for manual resizing. With Neon, users can create isolated copies of their data in seconds, reducing waiting times for clusters, databases, and colleagues' changes to deploy. The platform is designed to take the overhead out of starting, stopping, creating, and cloning databases from hours to seconds, focusing on providing everything Postgres developers need and less.

Neon has been operating its autoscaling feature for Postgres serverless instances for over a year, and it has seen significant adoption with almost a third of customers using it. The feature performs zero-downtime vertical scaling, provisioning extra CPU and memory as needed to handle peak demand without incuring peak infrastructure costs. Neon's architecture uses virtual machines in Kubernetes clusters, which provides strong isolation boundaries, dynamic resource adjustment, and seamless live migration. Despite some challenges, such as issues with cgroup signals and memory scaling, Neon has continued to prioritize stability and improve its autoscaling algorithm. Future improvements include expanding the set of workloads that autoscaling is a good fit for, smarter autoscaling using local file cache metrics, accommodating rapid memory allocation with swap, and enhancing observability into swap usage.

Neon offers a streamlined approach to using Postgres, enhancing developer productivity by abstracting away complex infrastructure tasks traditionally associated with database management, such as setup, connection pooling, and autoscaling. It introduces a serverless, scale-to-zero architecture that dynamically adjusts resources based on demand, significantly reducing costs and complexity. Neon supports innovative features like database branching, which allows developers to clone databases quickly for testing and prototyping without impacting the production environment. This capability, akin to Git for Postgres, empowers developers to experiment and iterate rapidly, improving development cycles and data recovery processes. Additionally, Neon's integration with existing development tools and serverless environments ensures a seamless experience, enabling developers to focus on building scalable and reliable applications without being encumbered by the intricacies of traditional database management.

Neon, a new approach to database development using Postgres, is now generally available, featuring significant improvements in database scaling and developer workflow efficiency. By separating storage and compute, Neon allows for sub-second cluster creation, automatic scaling, and instant branching, enabling developers to work independently without disrupting team collaboration. This innovative architecture, which remains open-source and compatible with existing Postgres tools and extensions, addresses traditional challenges of scaling and restoring relational databases by facilitating quicker recovery and reducing overhead. Neon's capabilities, such as serverless autoscaling and seamless database branching, are designed to enhance developer productivity by minimizing wait times and simplifying operations. These developments have been integrated upstream into the Postgres community, emphasizing Neon's commitment to evolving the Postgres experience while maintaining its core strengths.

Neon's implementation of storage sharding for Postgres aims to enhance scalability by separating storage and compute, with compute handled by virtual machines and pods that autoscale on-demand. The storage component uses a Rust-based, multi-tenant engine called Pageserver to store historical data, with sharding distributing data across multiple Pageservers to increase capacity and throughput. This approach prevents large and small databases from competing for resources on the same server and offers linear scalability of storage I/O with dataset size. Storage sharding is especially beneficial for applications requiring high storage capacity and read-intensive workloads. Neon's architecture allows for the storage of tens to potentially hundreds of terabytes, with plans for future optimizations like shard splitting and enhancing shard subscription to Safekeepers. This system is poised to support even larger datasets and higher throughputs, marking a significant step toward achieving bottomless storage.

Neon has introduced an innovative autoscaling feature for serverless Postgres that allows seamless scaling of CPU and memory resources without dropping connections or interrupting queries, thus enabling applications to handle peak demand efficiently while reducing infrastructure costs. Over a year after its launch, nearly a third of Neon's customers utilize this feature, demonstrating its effectiveness and stability, despite challenges such as kernel panics and I/O throttling. The architecture utilizes virtual machines in Kubernetes clusters, with a custom resource definition and controller called NeonVM, and an autoscaler-agent to monitor metrics and make scaling decisions. Recent improvements have focused on smarter autoscaling using metrics from the Local File Cache and accommodating rapid memory allocation through swap, with plans to enhance the scaling algorithm using HyperLogLog estimators. These advancements are part of Neon’s broader efforts to create a cloud-native serverless Postgres platform that separates storage and compute, offering unique features like branching and point-in-time restore. Neon encourages user feedback and participation to further refine its platform and support the development of next-generation applications.

This guide provides a step-by-step process for building user authentication APIs and HTML pages in an Astro application using Lucia Auth and Postgres powered by Neon. The process covers provisioning a serverless Postgres database, creating a new Astro application, setting up the database connection and schema, enabling server-side rendering with Vercel, defining the Astro application routes, building the user authentication routes, deploying to Vercel, and testing the authentication setup locally. By following this guide, users can implement user authentication in their Astro applications using Lucia Auth and Postgres powered by Neon.

The guide offers a comprehensive walkthrough for implementing user authentication in an Astro application using Lucia Auth and a serverless Postgres database powered by Neon. It details the necessary prerequisites, including Node.js, Neon, and Vercel accounts, and outlines the step-by-step process of setting up the database, configuring the Astro application, and integrating Tailwind CSS for styling. The guide further explains how to enable server-side rendering with Vercel, establish a secure Postgres database connection, and set up user authentication routes using Lucia Auth. Additionally, it covers the creation of HTML pages for user sign-up and sign-in, along with API routes to handle authentication logic. The document concludes with instructions for deploying the application to Vercel and highlights the importance of creating protected routes accessible only to authenticated users, emphasizing the security and scalability benefits of this setup.

This guide provides a step-by-step process for building an e-commerce store UI in Astro and Postgres using Neon and AWS Amplify. The prerequisites include Node.js 18 or later, a Neon account, and an AWS account. The steps involve provisioning a serverless Postgres database powered by Neon, creating a new Astro application, setting up a Postgres database connection and schema, defining the Astro application routes, enabling server-side rendering in Astro with AWS Amplify, and deploying to AWS Amplify. Throughout the guide, you'll learn how to integrate Tailwind CSS for styling, create mock data, populate the products table, define routes, and deploy your website to AWS Amplify. By following this guide, you'll be able to build a dynamic e-commerce store UI with Astro and Postgres using Neon and AWS Amplify.

This guide provides a comprehensive tutorial on building a dynamic e-commerce store UI using Astro, Neon Postgres, and AWS Amplify. It details the process of setting up a serverless Postgres database with Neon, creating an Astro application, integrating Tailwind CSS, and enabling server-side rendering with AWS Amplify. The guide walks through configuring a secure database connection, creating and populating a database schema, and defining application routes for product listing and display pages. It also covers deploying the application to AWS Amplify, including setting up environment variables and managing deployment settings. The tutorial is designed for users with some technical background, requiring Node.js, a Neon account, and an AWS account to follow along.

This post explains how to create a global serverless Nuxt SSR application that leverages Cloudflare Hyperdrive and Neon Postgres. The architecture uses a client request to make a query to the Neon database via a Cloudflare Worker configured with the Hyperdrive binding, which then executes or returns a cached response from Hyperdrive. This setup optimizes access to data globally, regardless of user location. The solution utilizes Drizzle ORM to connect to Neon databases and provides two integration options: using the Neon Serverless driver directly in Nuxt server routes or deploying a Worker with the Hyperdrive configuration accessible through the Drizzle Postgres HTTP Proxy driver. The post assumes familiarity with Cloudflare, Drizzle, and Nuxt and requires a Neon Postgres account and a Cloudflare Workers Paid plan for access to Hyperdrive functionality.

Here's a neutral and interesting summary of the text in one paragraph: Scale-to-zero databases allow developers to spin up instances as needed, reducing costs by shutting off idle compute resources while storing data safely. This feature is often misunderstood due to misconceptions about production databases being always-on, but it's actually beneficial for development teams who need temporary or low-usage databases. With scale-to-zero, developers can freely create and test databases without worrying about runaway costs, enabling more efficient experimentation, iteration, and testing of applications. It also removes the headache of database management and aligns costs with actual usage across the entire database fleet. By adopting scale-to-zero, teams can reduce costs, automate tasks, and offer more customization options to customers, ultimately moving faster without wasting money on idle databases.

Neon promotes its "scale to zero" database feature, which allows databases to shut down and incur no compute costs during periods of inactivity while maintaining data storage separately. This feature is particularly beneficial for developers who frequently spin up temporary databases for testing, prototyping, or analysis, as it aligns costs with actual usage and reduces the overhead of managing idle databases. Critics argue that production databases are always active, thus making scale-to-zero irrelevant for real workloads; however, Neon contends that most non-production databases are ephemeral and have idle periods, making this feature cost-effective and efficient for various development environments. Moreover, the analogy of German subway escalators that conserve energy when not in use is employed to illustrate that scale-to-zero databases do not compromise performance when active. Neon enables users to experiment, iterate, and test applications with minimal cost, allowing for more flexible and innovative database strategies, such as creating customer-specific databases with enhanced data isolation and security. By offering a free tier, Neon aims to lower overall costs while supporting a large number of databases, making it an attractive option for both small projects and large-scale operations.

The article details the process of building and deploying a global serverless Nuxt SSR application using Cloudflare Hyperdrive and Neon Postgres. The project involves deploying a Nuxt fullstack application on Cloudflare Pages, utilizing server routes to access a Neon Serverless Postgres database optimized by Cloudflare Hyperdrive for global data access. Hyperdrive accelerates query processing by managing a globally distributed pool of database connections, while Neon offers automatic scaling of compute resources based on demand. The application uses the Drizzle ORM Postgres HTTP Proxy to connect through a Cloudflare Worker configured with Hyperdrive, allowing flexible integration of dependencies. The guide provides step-by-step instructions on setting up the Nuxt app, configuring Hyperdrive and the Drizzle ORM, and deploying the app, highlighting the benefits of reduced latency and enhanced global responsiveness. The integration of technologies like Nuxt, Drizzle, and Hono simplifies development, and the article encourages users to explore Serverless Postgres with Cloudflare infrastructure.

Neon is a serverless Postgres database that simplifies operations through robust API support and database branching capabilities. To optimize Neon's performance, it's essential to understand compute sizing and connections management, including connection pooling, autoscaling, and planning resources based on expected concurrent connections. Troubleshooting slow query performance involves analyzing pg_stat_statements and EXPLAIN ANALYZE output to identify problematic queries, which can be optimized by adding new indexes, reducing bloat, caching data, and optimizing ingestion rates through strategies like client-side resource optimization, read replicas, batch insert operations, and regular index maintenance. Additionally, Neon's database branching feature allows for safely testing performance improvements in a production-like environment without disrupting live operations.

Neon Postgres, a serverless version of the Postgres database, leverages elastic scaling to optimize performance and simplify operations through robust API support and database branching capabilities. To maintain optimal performance, it is crucial to manage compute sizing effectively, utilize connection pooling, and leverage autoscaling to handle varying workloads. For troubleshooting query performance, tools like pg_stat_statements and EXPLAIN ANALYZE are essential in identifying and optimizing slow queries, while tactics such as adding indexes, managing table bloat, and improving cache usage further enhance efficiency. Ingestion rates can be improved by ensuring adequate client-side resources, using read replicas, executing batch insert operations, and periodically cleaning up unnecessary indexes. Neon’s database branching allows for safe testing of performance improvements without impacting production databases, enabling users to experiment freely with different configurations.

Building a Next.js application with Clerk for authentication and Neon's Serverless Postgres with Drizzle ORM requires careful consideration of compliance with standards such as SOC 2 and HIPAA, multi-factor authentication, and user management. Using Clerk enables seamless addition of advanced authentication and user management features to the application while complying with data security regulations by storing personally identifiable information in their compliant infrastructure. The application uses a complete example architecture that includes Vercel serverless hosting, Next.js application direction for users to Clerk for authentication, Clerk redirection back to the Next.js application after successful authentication, and Neon's Serverless Postgres database management using Drizzle ORM. By following this guide, developers can build an effective authentication and authorization system into their application while leveraging the benefits of Clerk and Neon's services.

The article provides a detailed guide on building a Next.js application that integrates Clerk for authentication and Neon's serverless Postgres database using Drizzle ORM. It emphasizes the importance of utilizing existing authentication solutions like Clerk to streamline user management features and comply with security standards. The guide walks readers through setting up a Next.js project, configuring Clerk for user authentication, and using Drizzle ORM to manage data within a Neon-hosted Postgres database. It includes instructions for setting up the development environment, creating schema migrations with Drizzle Kit, and implementing user actions for storing and managing data associated with user IDs. The article culminates in a practical example that allows users to save and display personal quotes, demonstrating the seamless integration of authentication and data storage in a modern web application.