October 2024 Summaries

Scott Cate, founder of the new link shortener platform 301.Pro, shares his experience transitioning from AWS RDS Postgres to Neon for backend analytics. He chose Neon due to its simplicity and cost-effectiveness compared to AWS RDS. The blog post series will cover the process of building the backend analytics for 301.Pro using Neon, including advanced features like branching and configuring projects and branches. The platform offers unique features such as geolocation-based redirects and time-sensitive rules, making links smarter and more efficient for campaigns.

Scott Cate, the founder of 301.Pro, a link shortener platform, discusses his decision to transition from AWS RDS to Neon for backend analytics due to Neon's simpler and more efficient user experience. 301.Pro enhances traditional link-shortening with advanced features like geolocation-based redirects, time-sensitive rules, and dynamic UTM source adjustments, providing users with a smarter and more customized link management experience. The backend is built on a multi-cloud, serverless architecture using AWS Lambda and Cloudflare Workers, delivering fast results and personalized experiences. Neon's integration into this setup has been seamless, offering efficient analytics collection and connection pooling, although Cate advises against using pooling for long-running operations like database migrations. As he continues to explore Neon's advanced features, such as branching workflows, Cate considers how to optimize data isolation and performance as 301.Pro expands.

Neon Authorize is a new tool that simplifies the usage of Postgres row-level security (RLS) policies, enabling developers to manage RLS directly in their codebase and integrate with any authentication provider. This feature allows for fine-grained access control without added complexity. With Neon Authorize, calls to the database can be authenticated with a JWT generated by the auth provider, which is verified by the Neon Proxy and made available for RLS rules and WHERE clauses alike. The tool also offers utility functions such as auth.session() and auth.user_id(). In partnership with Drizzle ORM, developers can set up RLS policies in a declarative format right alongside their schema definitions, making the codebase more organized and easier to maintain.

Neon RLS is a new tool integrated into the Neon Data API, designed to simplify the implementation of row-level security (RLS) policies in PostgreSQL databases. It allows developers to manage RLS directly within their codebase and integrate with any authentication provider, thereby facilitating fine-grained access control with less complexity. By leveraging JSON Web Tokens (JWTs), Neon RLS enables authenticated database requests, which can be used in conjunction with RLS rules to enhance security. The tool supports integration with various authentication providers and offers a partnership with Drizzle ORM for a more intuitive setup of RLS policies. While it is recommended to use RLS for critical authorization logic, especially in multi-tenant enterprise applications, the decision to apply RLS universally remains a subject of debate. Neon RLS aims to make application development faster and more secure by allowing client-side applications without the need for a server or backend, although a backend is still advised for more substantial projects.

This post explains how to set up nightly backups for multiple Neon Postgres databases using GitHub Actions and pg_dump/restore. The process involves three main components: AWS, Postgres Database, and GitHub Action. The author demonstrates the setup with a single S3 bucket holding multiple folders, each containing backup files created by a GitHub Action. The code discussed in this post covers a single Action, but the same method can be applied to multiple Actions. The GitHub repository secrets are used for secure access and interaction with AWS services. The steps involved include installing PostgreSQL, configuring AWS credentials, setting file, folder, and path variables, creating folders if they don't exist, running pg_dump, emptying the bucket of old files (optional), and uploading to the bucket.

The article outlines a method for setting up automated nightly backups of multiple Neon Postgres databases to AWS S3 using GitHub Actions and the pg_dump/restore utility. It details the configuration of AWS, Postgres database connection strings, and GitHub Actions permissions necessary for the process. The approach involves configuring a single S3 bucket to store backup files in separate folders, each managed by an individual GitHub Action with its own schedule and retention period. The process includes installing PostgreSQL, configuring AWS credentials, setting file and folder variables, running the pg_dump command, and managing the retention of old files. The article emphasizes the importance of ensuring correct database URLs to avoid misplacing backups and advises on handling GitHub Actions timeouts by potentially using self-hosted runners.

This post guides users on how to create an AWS S3 bucket for storing Postgres backups. It is the first part of a two-part series on automating nightly backups for multiple Neon projects, which can be useful in multi-tenant architectures with one project per customer. The process involves setting up an OIDC Identity Provider, creating a role, and creating an S3 bucket with its corresponding policy. These steps are crucial to enable the use of AWS credentials by the GitHub Action for backing up data from Neon projects to S3 every night.

The blog post presents a detailed guide on setting up an AWS S3 bucket for storing Postgres backups, specifically in the context of automating nightly backups for multiple Neon projects. The article outlines the process as part of a two-part series, focusing on configuring AWS components, such as creating an OIDC Identity Provider to authenticate GitHub Actions and establishing an AWS Role to delegate permissions for accessing AWS services. It also explains creating an S3 bucket, updating its policy to allow specific actions by the Role, and highlights the importance of managing geographical regions to optimize backup efficiency. The post concludes by emphasizing the use of GitHub Secrets to securely store sensitive information like AWS account details and bucket names, and introduces Neon as a serverless platform for Postgres with features like autoscaling and database branching.

High availability (HA) in managed databases refers to the system's ability to remain operational and accessible "for a high percentage of time," even during failures. Neon, a managed database service, ensures HA by separating storage and compute layers. Compute failures are resolved quickly as stateless processing nodes can be replaced immediately. Storage is designed with redundancy and ultra-high durability through the use of cloud object storage, automatically replicating and distributing data across multiple nodes for failure tolerance. Neon's Safekeepers handle Write-Ahead Log (WAL) replication with redundancy across multiple AZs, while Pageservers act as a fast-access disk cache for compute. The primary copy of all data resides in cloud object storage, providing ultra-high durability.

High availability (HA) in managed databases ensures systems remain operational even during failures, though its exact definition varies by vendor. Neon approaches HA by separating storage and compute, thereby allowing quick resolution of compute failures through the immediate provisioning of new, stateless compute nodes. Storage, designed with high durability and distributed across multiple Availability Zones, uses a combination of Safekeepers for Write-Ahead Log replication, Pageservers for fast data access, and cloud object storage for long-term data security. This architecture guarantees data safety and operational continuity, with redundancy and fault tolerance built into all plans, including free ones, ensuring that Neon maintains a robust and reliable database service.

BeatGig, a marketplace that streamlines talent booking process, uses Neon's read replicas to provide easy, read-only access to its team while minimizing maintenance efforts. Neon stands out for its fast provisioning, query speed, native connection pooling, and ease of use even for non-SQL-experts. The platform's read replicas are lightweight, quick to deploy, and cost-efficient as they share the same storage backend with primary compute and scale to zero when not in use. Neon has made these read replicas available in its Free Plan for users to test them out.

BeatGig, a marketplace connecting artists, managers, and venues, has successfully integrated Neon's read replicas to enhance its operations by providing read-only data access without impacting production. Neon was chosen over other SQL providers for its fast provisioning, efficient query speed, and ease of use, particularly benefiting those without extensive Postgres or DevOps expertise. The read replicas allow BeatGig's team to run business intelligence and analytics queries safely, as they can be spun up quickly, require no additional storage, and scale to zero when not in use, thus being cost-effective compared to traditional options like AWS RDS or Aurora. This approach safeguards the primary compute from inefficient queries, offering a more streamlined and user-friendly experience for BeatGig's developers and non-SQL experts. Neon's solution, now available in their free plan, allows users to experiment with read replicas without financial commitment, further encouraging adoption and exploration.

Neon has introduced read replicas for its Free Plan, allowing users to test their capabilities. Read replicas are a way to use compute hours and can be used for offloading reporting and analytics tasks from the primary database. Unlike traditional setups, Neon's read replicas don't need to copy or duplicate data, making them lighter and cost-efficient. They also scale to zero when idle, saving costs and easing maintenance pressure. Key advantages of Neon's read replicas include lightweight nature, worry-free management, and fast deployment.

Neon has expanded its Free Plan to include read replicas, which allow users to efficiently utilize their 190 CU-hours by offloading reporting and analytics tasks from the primary database without duplicating data. Unlike traditional read replicas, Neon's unique serverless architecture enables these replicas to read directly from a shared storage backend, making them lighter, cost-efficient, and fast to deploy. This setup allows users to create numerous replicas with minimal cost, as they scale to zero when idle. Popular use cases for Neon's read replicas include providing safe, low-cost read access for team members and facilitating horizontal scaling by offloading read queries from the primary compute. To create a read replica, users simply navigate to the desired branch in their Neon Free account, configure the replica, and use a dedicated connection string to access it.

Event-driven architectures using Neon and Sequin provide an alternative solution for capturing changes from your database (CDC) without requiring significant time and expertise to properly configure new infrastructure like Debezium and Kafka. Sequin is a tool that detects, streams, and processes every row change in Postgres databases such as Neon. It uses logical replication slots to create a strictly ordered view of rows in your tables, making it possible to process every change to a table's rows as a stream. This setup allows developers to consume rows using Sequin's APIs without copying the data to a new service.

Event-driven architectures using Neon and Sequin offer a streamlined approach to capturing and processing database changes without the complexity of traditional tools like Debezium and Kafka. By utilizing Sequin, changes in a Postgres database can be detected and streamed in real-time through logical replication slots, enabling immediate processing by consumers. This approach allows for the streaming of database changes directly from Neon without duplicating data, leveraging Neon's developer tools such as branching and recovery. The setup involves connecting a Neon database to Sequin, configuring tables for streaming, and creating consumer groups to process changes, all while ensuring exactly-once processing guarantees. The integration simplifies the infrastructure required for change data capture, making it accessible through Sequin's APIs and avoiding the need for additional dependencies.

Isolated application environments, as opposed to a shared environment, involve deploying and managing a separate system for each customer, offering advantages such as meeting geographical and political data residency requirements and allowing customers to control upgrade schedules. While this approach is not multitenancy, a well-equipped control plane can manage customer deployments similarly. However, it requires careful consideration due to its increased operational overhead and the temptation to create bespoke customizations for individual customers, which can complicate software maintenance. Managing isolated environments involves provisioning, monitoring, and maintaining each instance from a central control plane. Independent versioning per customer is a primary benefit but necessitates strict discipline and careful management of software development lifecycles to ensure reliable upgrades and minimize possible paths. This approach, while resource-intensive, can provide significant flexibility and benefits for specific customer needs, particularly those requiring custom geographic deployment and version management options.

Isolated application environments, as explored in a blog post by Dian M Fay, offer a tailored approach to database-per-user architecture by providing separate systems for each customer, as opposed to a shared environment. This method addresses specific needs such as compliance with data residency laws, allowing customers to dictate upgrade schedules, and accommodating geographical preferences. However, it requires significant operational overhead, as each customer's system must be uniquely managed, provisioned, and maintained, often involving complex version management and a central control plane. Although this setup can enhance flexibility and cater to individual customer demands, it also introduces challenges in maintaining uniformity across deployments. The article underscores that while isolated environments offer distinct benefits, they demand careful consideration and substantial resources, making them suitable only under certain conditions. This piece is part of a series examining database-per-user architectures, with previous parts focusing on multi-tenancy, control planes, and shared application environments.

The article explores the design and management of a shared application environment in a database-per-user architecture, a classic SaaS model where a single application environment is shared among all users. This approach offers simplicity and operational efficiency, as there's just one system to manage initially, which can be attractive for smaller engineering teams. However, it brings challenges like maintaining consistent schema versions across user databases and navigating data residency issues without geographic constraints. The shared-environment strategy emphasizes the importance of a central catalog database for managing user data and routing queries, simplifying the security model by centralizing access decisions. It also highlights the potential for operational complications, such as schema upgrades, which require careful planning and testing to avoid disruptions. The article suggests that while the shared environment simplifies some aspects, it introduces complexities that must be managed, making it suitable for certain scenarios but not all. This piece is part of a broader series on database-per-user architectures, with future articles set to discuss isolated application environments.

Deploying a shared application environment for a database-per-user architecture simplifies the operational aspects of SaaS applications but comes with trade-offs in user flexibility and data management. This approach involves deploying a single application environment shared among users, which offers simplicity and flexibility but requires a consistent schema version across all user databases, impacting the ability to stay on specific software versions. User authentication and data routing are centralized in a catalog database, which tracks user membership and database connections, ensuring that user requests are directed to the correct database. While this architecture eases maintenance by reducing the number of components needing updates, it requires careful planning for schema changes and can limit user control, especially when dealing with geographical or data residency constraints. The shared environment model is contrasted with isolated environments, which will be explored in the next article of the series.

Mindvalley, a global platform for personal growth and transformation, has minimized its time-to-launch by using Neon branches. Previously, the company used Google CloudSQL for their production databases but experienced inefficiencies when handling development and testing within this environment. By leveraging Neon's faster DX for development and testing while keeping production databases untouched, Mindvalley has significantly reduced waiting times and sped up its development cycle. The key feature driving this boost in developer velocity is branching, which allows developers to spin up new previews in less than a minute. Automatically creating branches with every PR has drastically reduced waiting times, speeding up their development cycle.

Mindvalley, a global platform focused on personal growth, has streamlined its development process by integrating Neon for non-production workflows, significantly reducing time-to-launch for new features. Previously reliant on Google CloudSQL for both production and development environments, Mindvalley faced inefficiencies such as concurrency issues and prolonged end-to-end testing times. By implementing Neon, they have leveraged database branching to allow developers to quickly spin up independent development environments, thereby avoiding bottlenecks associated with shared instances. The Neon Twin workflow, which involves nightly data synchronization via GitHub Actions, enables Mindvalley to maintain up-to-date testing datasets without manual intervention, further enhancing developer productivity. This setup allows for rapid deployment of new features while maintaining the integrity of the production database, addressing previous challenges related to database seeding and testing environment maintenance.

Many Postgres users do not upgrade to the latest version due to two core issues: Postgres works well and upgrades are difficult. While older versions of Postgres are capable for most developers' needs, newer versions offer significant performance improvements and new features. However, major updates can be time-consuming and resource-intensive, leading many organizations to postpone upgrades until necessary. Despite the challenges, it is recommended to upgrade regularly to benefit from improved functionality and compatibility with new tools and frameworks.

While Postgres 17 has introduced significant performance improvements and features, many users remain on older versions due to the inherent stability and robustness of Postgres, which meet most developers' needs without requiring frequent upgrades. Upgrading major Postgres versions can be challenging due to potential backward compatibility issues and the extensive effort required, particularly for large databases. Companies like Knock and Retool have successfully navigated such upgrades by meticulously planning and implementing strategies like logical replication and thorough testing. Despite the challenges, upgrading is encouraged to avoid future technical debt and to take advantage of new functionalities and performance enhancements. Tools like pgversions.com can help users identify what they are missing by not upgrading and guide them through the upgrade process. Neon aims to simplify upgrades by introducing features like one-click upgrades and providing robust support for Postgres 17, encouraging users to stay current and avoid the complications of leaping multiple versions at once.

Deno, a JavaScript/TypeScript runtime built by Node.js creator Ryan Dahl, has been successful in its mission to be more secure and simpler than Node. However, it has struggled with widespread adoption due to the lack of compatibility between the two ecosystems. With Deno 2.0, this is changing as it brings Node.js and npm compatibility directly into Deno, allowing developers to use both platforms simultaneously. This version also introduces long-term support (LTS), which provides stability and predictability for large-scale projects and enterprises, making Deno a more viable option for these organizations.

Deno 2.0 marks a significant evolution in the JavaScript runtime landscape by introducing backward compatibility with Node.js and npm, addressing previous adoption barriers despite its enhanced security and simplicity over Node.js. Developed by Ryan Dahl, Deno initially aimed to rectify what he perceived as Node.js's complexities and security issues by aligning more closely with ECMAScript standards and simplifying dependency management through direct URL imports. The new version builds on this foundation by allowing seamless interoperability with Node.js infrastructure, supporting built-in Node.js modules, and enabling the use of npm packages within Deno projects. This allows developers to leverage familiar Node.js tools while benefiting from Deno's security model and modern JavaScript features. Deno 2.0 also introduces long-term support (LTS) channels, providing stability and predictability for large-scale projects and enterprises, thus making it a more viable option for production environments. By offering these enhancements, Deno 2.0 aims to facilitate a gradual transition for developers who wish to explore its advantages without abandoning the well-established Node.js ecosystem.

Dispatch, a technology company that streamlines last-mile delivery for businesses, has adopted serverless writer endpoints from Neon to alleviate overprovisioning and accelerate development. The company uses Neon branches for their ephemeral environments, allowing them to create isolated testing environments without moving large amounts of data around. Dispatch synchronizes their development environments with a subset of workloads on a scheduled basis using AWS Database Migration Service (DMS) and creates isolated branches for each environment through Kubernetes integration. Neon's read replicas are also used for ETL and analytics, improving the efficiency of data pipelines.

Dispatch, a technology company focused on improving last-mile delivery, has started leveraging Neon to enhance its development processes and tackle challenges posed by its existing AWS Aurora infrastructure. With operations in over 75 U.S. markets, Dispatch traditionally faced issues with overprovisioning Aurora to manage spiky traffic patterns, resulting in inefficiencies. By utilizing Neon’s serverless writer endpoints and branching model, Dispatch can create isolated, ephemeral environments that simplify data management and testing, reducing operational burdens and costs. This transition supports their move towards a microservices architecture, enabling more agile workflows by allowing realistic data replication for testing without the need for extensive data transfers. The company integrates Neon with Kubernetes to automate environment creation, and Neon’s read replicas are used to improve the efficiency of data pipelines, particularly for ETL and analytics tasks.

Starting from October 10th, Neon has increased the project limit on its Free Plan from one to ten projects. This change is expected to improve the quality of life for Free Plan users by allowing them to learn new stacks, try Azure, launch an AI-powered IDE, or ship a minimum viable product (MVP). Despite concerns about the financial viability of database free tiers, Neon's unique architecture allows it to offer these benefits at a significantly lower cost compared to other architectures. This is due to the separation of storage and compute, which enables scale-to-zero capabilities and lowers the minimum cost of running a single database on Neon.

Neon has expanded its Free Plan, allowing users to create up to ten projects, responding to feedback for more flexibility. This enhancement does not change account-level limits such as compute hours, storage, and bandwidth. Most Free Projects operate with minimal resources, while those needing more typically transition to paid plans. Neon focuses on reducing operational costs by employing a separation of storage and compute, utilizing local storage for hot data and S3 for cold data, resulting in significantly lower costs than traditional architectures. This cost-efficient design supports innovative uses, such as powering AI platforms and managing vast numbers of databases for companies like Retool, and is consistent with Neon's founding vision of creating a modern, efficient Postgres service for the cloud.

The first Azure region for Neon projects is now available in East US 2, marking the initial milestone on their Azure roadmap. Deeper integrations with the Azure ecosystem are expected soon. Users can create their first Neon project on Azure by selecting "Azure" as their cloud provider. More regions will be added based on user feedback. During the beta phase, users won't be charged for exceeding plan usage limits and regular SLA does not apply. Feedback is welcome to shape the future of Neon on Azure.

Neon has announced the beta availability of its first Azure region, East US 2, marking a significant step in its integration with Microsoft's cloud platform. Users can now deploy Neon databases in this region and will soon have access to more Azure regions as part of Neon's ongoing roadmap. During the beta phase, which will last a few weeks, users on Launch, Scale, and Business plans can test the service without additional costs beyond their plan's usage limits. However, it is advised to use the service for testing and prototyping rather than critical workloads, as the regular service level agreement does not apply. Neon is also available as a native integration in the Azure Marketplace, and the company is actively seeking user feedback to enhance its Azure offerings. The beta launch signifies the start of a deeper partnership with Microsoft and an effort to more tightly integrate Neon within the Azure ecosystem.

Neon has achieved ISO 27001 and ISO 27701 compliance, adding to its existing list of compliances including GDPR, CCPA, and SOC 2 Type 2. These certifications demonstrate the company's commitment to providing industry-leading security and privacy for customers. Neon is setting a new benchmark in the database platform industry by exceeding compliance standards and consistently preparing for future challenges. The company plans to work towards achieving HIPAA compliance in 2025, further strengthening its ability to handle sensitive healthcare data.

Neon has achieved ISO 27001 and ISO 27701 compliance, adding to its existing certifications, including SOC 2 Type 2, GDPR, and CCPA, reinforcing its commitment to leading the industry in security and privacy for database platforms. ISO 27001 focuses on an information security management system, while ISO 27701 extends to privacy information management, emphasizing the handling of personal data in compliance with global standards. The company underwent a successful year-long preparation and audit process, demonstrating its readiness and commitment to high security standards. By exceeding compliance requirements, Neon positions itself ahead of competitors, providing robust data protection and setting a higher industry standard. Looking ahead, Neon aims to achieve HIPAA compliance in 2025 to enhance its capability in managing sensitive healthcare data, reflecting its continuous dedication to improving security and privacy measures.

The combination of Neon and ClickHouse can provide a powerful solution for real-time analytics on transactional data. Neon is a serverless Postgres service that excels at handling transactional workloads, while ClickHouse is a high-performance columnar database optimized for real-time analytics. By using PeerDB to enable Change Data Capture (CDC) from Neon to ClickHouse, developers can easily sync data and power real-time analytics and data warehousing use cases. This combination allows businesses to keep their data up-to-date and accessible for analytics and decision-making while maintaining operational efficiency with large-scale analytics.

The text discusses the integration of Neon, a serverless Postgres service, and ClickHouse, a high-performance analytical database, for real-time analytics on transactional data. It highlights the benefits of using Neon for low-latency transactional applications and ClickHouse for fast analytical processing. PeerDB, a provider of Change Data Capture (CDC) solutions, facilitates data replication from Neon to ClickHouse, enabling real-time data synchronization and analytics. The acquisition of PeerDB by ClickHouse enhances this integration, with PeerDB's technology offering optimized replication performance. The text outlines the process of configuring Neon and ClickHouse for data synchronization using PeerDB, emphasizing the ease of setup and the advantages of combining operational efficiency with large-scale analytics.

Neon has increased the number of projects included in its paid plans - Launch, Scale, and Business. The new Business plan is designed for production workloads, offering higher resources, migration assistance, and premium technical support. Pricing changes include increasing the Launch Plan from 10 to 100 projects, the Scale Plan from 50 to 1,000 projects (with an extra 500 available for $50/month), and the Business Plan with 5,000 projects at no additional cost (with an extra 5,000 available for $50/month). The new Business plan also includes features like shared storage, more compute usage, priority support, migration assistance, organization accounts, branch protection, instant PITR and time-travel queries, compliance, and security.

Neon has updated its pricing plans to include a significantly higher number of projects without increasing costs, with the Launch Plan now supporting 100 projects, the Scale Plan 1,000 projects, and the new Business Plan offering 5,000 projects. Designed for diverse use cases such as SaaS with database-per-user and developer platforms, Neon's serverless Postgres solution offers advantages in scalability and cost-efficiency, particularly benefiting applications with variable traffic and dev/test workflows. The Business Plan, tailored for production workloads, provides 500 GB of shared storage, enhanced compute usage, priority support, migration assistance, organization accounts, and security features like branch protection and instant point-in-time recovery. This strategic expansion aims to facilitate database-per-tenant architectures and cater to teams requiring robust production database solutions, with an emphasis on ease of migration and operational efficiency.