March 2024 Summaries

Retool, a platform that enables developers to create internal apps for their teams and businesses, uses Neon API to manage over 300k Postgres databases with just one engineer. The integration of Neon's serverless Postgres database allows Retool to automate virtually all database management tasks, streamlining workflows and reducing manual work. This partnership has enabled Retool to offer managed Postgres to their users while maintaining cost-efficiency and operational sustainability.

Retool utilizes the Neon API to efficiently manage over 300,000 Postgres databases with minimal engineering resources, demonstrating the potential of serverless architecture and automated database management. By integrating Neon's serverless Postgres solutions, which automatically scale to zero when inactive, Retool minimizes idle resource costs and manages a vast database fleet with just one engineer. Retool's platform, known for enabling rapid internal app development through its drag-and-drop interface, seamlessly integrates these databases, allowing developers to focus on building custom internal tools without the burden of database management. Retool's internal operations are enhanced by using its own platform to oversee database tasks, showcasing the benefits of their approach to potential partners and customers interested in offering managed Postgres solutions.

White Widget is an award-winning software agency that uses Neon as its backend for various applications, citing its flexibility and smooth developer experience as key benefits. The company previously relied on managed Postgres instances from GCP's Cloud SQL but found the service inflexible and expensive. White Widget sought a scalable Postgres solution with dynamic scaling capabilities and usage-based pricing. They found Neon's architecture of separating storage from compute, supporting serverless scalability, and connecting multiple ephemeral compute endpoints to a single storage entity, to be particularly appealing. This allows for autoscaling, read replica management, and cost-efficient performance optimization. With Neon, White Widget can dynamically scale its compute resources in response to fluctuating application demands, ensuring optimal performance and cost-efficiency without overprovisioning or underutilization.

White Widget, a leading software agency known for creating user-centric web and mobile applications, has optimized its use of Postgres by adopting Neon, a serverless database architecture that separates storage from compute to enable dynamic scalability. Initially using GCP's Cloud SQL, White Widget encountered issues with resource allocation, either overspending or facing shortages during peak times. Neon’s architecture offers a solution by allowing resources to scale automatically in response to demand, thus enhancing performance and cost efficiency. This approach suits White Widget's diverse projects, including AI solutions and ride-sharing platforms, by providing a flexible and efficient backend. Neon’s ability to facilitate read replicas that scale down when not needed further contributes to the agency's streamlined operations. White Widget continues to explore new ventures, such as a productivity platform on the Matrix protocol, leveraging Neon's capabilities for future growth.

To optimize storage costs for Postgres databases, consider removing unused indexes that consume storage space and add overhead to write operations, reducing storage duplicities by using database branching or multiple data copies, using efficient data types such as SMALLINT instead of BIGINT, ordering columns from largest to smallest to minimize padding, adjusting the history retention period in Neon projects to reduce storage size, and fine-tuning autovacuum schedules to reclaim storage space after rows are updated or deleted. By implementing these strategies, you can streamline your storage usage and keep your Postgres database running lean, making it more efficient, agile, and cost-effective.

The text provides strategies for optimizing storage costs in Postgres databases by addressing issues such as unused indexes, storage duplicities, inefficient data types, column order, history retention, and autovacuum tuning. It highlights the importance of removing obsolete indexes to free up space and reduce write overhead, and suggests using efficient data types and rearranging column orders to minimize padding and wasted space. The text also discusses the benefits of using database branching, as provided by Neon, to reduce storage duplicities by sharing underlying data across environments, and adjusting the history retention period for cost savings while considering the trade-offs in recovery capabilities. Additionally, it emphasizes the significance of fine-tuning the autovacuum process to manage storage efficiently, particularly in highly transactional environments. Neon is presented as a tool that can enhance Postgres efficiency through its innovative storage management features, offering a free trial for potential users.

We’re Neon. We’re building Postgres that helps you confidently ship reliable and scalable apps. You can try Neon now for free. We recently added support for IPv6 addresses in the IP Allow feature. This post explains what IPv6 is and its benefits. IPv4 has been around for almost half a century, using 32-bit addresses which allowed for about 4.3 billion unique addresses, but this number is insufficient due to the explosive growth of the internet and smart devices. A solution was needed, and IPv6 was developed with 128-bit addresses that significantly expands the possible addresses to approximately 340 undecillion. This vast expansion solves the primary problem of IPv4 address exhaustion. Adding support for IPv6 addresses in IP Allow allows you to overcome IPv4 limitations, build highly scalable architectures while maintaining backward compatibility with existing IPv4 workloads. This is particularly useful for large-scale and containerized applications. Dual-stack devices will often attempt to connect using IPv6 first, but if the connection cannot be established over IPv6, it falls back to using IPv4.

Synthetic data can help developers catch more bugs before shipping to production by generating diverse, statistically identical, and structurally similar data sets that replicate the complexity of real-world data without containing sensitive or personal information. This allows for faster testing, automated testing, and performance testing, reducing privacy and security risks associated with using production data. By leveraging synthetic data tools and platforms, developers can increase confidence in their application's resiliency and catch bugs earlier in the development process.

Developers face increasing pressure to create robust, scalable, and secure applications, with synthetic data emerging as a pivotal solution for testing and bug detection. Unlike production data, which carries privacy risks and only reflects processed scenarios, synthetic data is generated using machine learning models and simulation algorithms to mimic real-world data statistically and structurally without sensitive information. This allows developers to explore diverse data sets, automate testing, and conduct performance assessments without security concerns. Tools like Neosync enable rapid data generation, enhancing test coverage and application resilience. As synthetic data tools continue to advance, they present a compelling option for developers aiming to improve their workflows and application reliability.

Neon, a company focused on building scalable and reliable PostgreSQL databases, has introduced support for IPv6 addresses in its IP Allow feature to address the limitations of IPv4, which is running out of available addresses due to its 32-bit structure. IPv6, with its 128-bit addresses, provides a vast number of unique addresses, solving the issue of address exhaustion and enabling scalable internet growth. The inclusion of IPv6 in Neon's IP Allow feature enhances data security by restricting database access to trusted IPs and supports the deployment of global environments with end-to-end IPv6 connectivity, as seen with cloud providers like AWS. This addition allows for the creation of scalable architectures while maintaining compatibility with existing IPv4 systems, particularly benefiting large-scale and containerized applications.

Prisma users can now run SQL migrations and scale their Postgres applications using Neon's default connection string, eliminating the need for direct database URLs, pooled connections, and shadow databases. The integration with Prisma allows developers to confidently ship reliable and scalable apps, taking advantage of type-safe queries and schema migration support via pooled connections. With the removal of manual database management and the introduction of new SQL commands, Neon's developer experience has been significantly improved, making it easier for developers to build the next generation of web applications.

Neon has enhanced its Postgres offering by integrating it with Prisma, an ORM that simplifies schema migrations and database object mapping for JavaScript and TypeScript applications. This integration now supports schema migrations via pooled connections, allowing the use of Neon's default connection string to scale serverless applications without needing multiple URLs or the "pgbouncer=true" parameter. Enhancements in PgBouncer 1.22.0 and Prisma Client 5.10.0 have resolved previous issues with prepared statements, enabling efficient schema migrations with a single pooled connection. Additionally, Prisma Migrate can now manage shadow databases more effectively, supporting DROP WITH (FORCE) to handle schema changes without manually creating shadow databases. These improvements aim to streamline the developer experience, particularly for serverless services, by simplifying connection management and enhancing scalability.

This guide explains how to integrate AWS Amplify Hosting CI/CD with Neon Postgres to test an application's environment or feature branches using real data enabled by Neon's database branching. The architecture automates the workflow and ensures each new feature or environment branch in your Amplify application has a corresponding, isolated database. This is especially useful for Server Side Rendering (SSR) applications built with frameworks like Next.js, Nuxt, Astro, and SvelteKit. Both Neon and Amplify Hosting are serverless, which means the underlying infrastructure doesn't require maintenance and scales down to zero when not in use. The integration involves a custom Bash script (neon-ci.sh) to manage Neon database branches alongside Amplify app branches.

Integrating AWS Amplify Hosting with Neon Postgres enables a seamless CI/CD process for server-side rendering (SSR) applications, allowing each application branch to have an isolated database branch. This architecture is particularly beneficial for frameworks like Next.js, Nuxt, Astro, and SvelteKit, providing an automated workflow that scales down when not in use, leveraging serverless compute functions and auto-scaling database clusters. The process involves setting up Amplify app branches that correspond to different environments and using a custom Bash script to manage Neon database branches, ensuring each branch has the correct database connection string for server-side environments. Neon’s serverless Postgres features, like connection pooling and database branching, allow developers to clone entire clusters quickly for testing without impacting the primary database, while the integration with Amplify supports streamlined deployment through a Git-based workflow. The setup requires configuring build settings in Amplify, using the Neon CLI for database management, and ensuring secure handling of API keys and permissions via AWS IAM roles and SSM Parameter Store, thus providing a robust solution for managing full-stack serverless applications.

The text discusses various methods of copying Postgres databases and introduces Neon branching as an alternative solution. Traditional methods such as physical file copy, logical backup and restore, logical replication, and database forking can be time-consuming and resource-intensive, especially for large databases. Neon branching utilizes the copy-on-write technique to create instant copies of Postgres databases without requiring additional storage until changes are made. This method is efficient, affordable, and allows developers to work on isolated database instances for testing, development, staging, point-in-time restores, and more.

Neon offers a groundbreaking solution for copying large Postgres databases through its innovative database branching technique, which enables instant and resource-efficient duplication by utilizing the copy-on-write method. This approach is particularly advantageous for creating multiple environments for testing, development, and point-in-time restores without the typical time and resource burdens of traditional methods like physical file copies, logical backups, or database forking. Unlike these conventional methods, which often require significant downtime and resource allocation, Neon's database branching allows for immediate copy creation and is cost-effective as it optimizes storage utilization by only allocating additional storage when the secondary branch diverges from the primary. This capability supports developers by providing isolated environments to work with real data, facilitating more efficient testing and development processes.

Autoscaling is crucial for applications like Recrowd, which experience fluctuating demand due to variable traffic patterns. Traditional databases often require manual provisioning or resizing, leading to overprovisioning, downtime, and cost inefficiencies. Neon's autoscaling capabilities dynamically adjust compute resources to match current demand, eliminating the need for manual intervention, minimizing downtime, and optimizing resource utilization. By leveraging autoscaling, Recrowd can ensure optimal performance during peak traffic periods, providing a seamless user experience and improving overall reliability.

Recrowd, Italy's leading crowdfunding platform, has successfully addressed its scalability challenges by utilizing Neon's autoscaling capabilities, transitioning from a manual, time-consuming database management approach with MySQL in Azure, which often resulted in downtime during high-traffic periods. Neon, a serverless Postgres solution, offers dynamic autoscaling that automatically adjusts compute resources according to demand, ensuring seamless operations without manual intervention or cost inefficiencies. This feature allows Recrowd to efficiently manage unpredictable traffic spikes during new investment project launches, maintaining optimal performance and an uninterrupted customer experience. The switch to Neon has eliminated the need for pre-provisioning or manual resizing, reducing downtime and allowing the Recrowd team to focus on providing investment opportunities with transparency and security for their users.