March 2023 Summaries

Bunnyshell has integrated Neon Postgres into its Environment-as-a-Service platform, enabling teams to create on-demand environments with production-like databases. This integration aims to increase developer productivity by providing realistic data and eliminating environment-related inconsistencies. Bunnyshell chose Neon due to its ability to provide isolated, production-like databases through its branching feature, which can automatically scale up or down based on demand.

Neon is a single writer, multiple readers, and multi-tenant system that separates compute and storage for efficient database management. It runs in Kubernetes containers and uses a non-overwriting storage engine to simplify setup and maintenance. The separation of compute and storage allows for serverless functionality, independent scaling, and integration with cloud storage like Amazon S3 or Google Cloud Storage. Neon's storage engine is modeled as a key-value store and supports time-travel queries and branches, eliminating the need for traditional backups. It also employs non-overwriting storage to replace traditional backups and work seamlessly with older versions of data.

Neon introduces a novel storage engine characterized by the separation of compute and storage, as well as a non-overwriting storage model, enhancing the traditional database architecture. This system, optimized for cloud environments and implemented within Kubernetes containers, allows for serverless operation, independent scaling of compute and storage, and efficient handling of multi-tenant workloads. Neon's storage engine operates as a key-value store and draws inspiration from Log-Structured Merge (LSM) trees, ensuring data immutability and seamless integration with cloud storage solutions like Amazon S3. The non-overwriting storage format eliminates the need for traditional backups, as it supports time-travel queries and maintains historical data access through a combination of delta and image layers. This design allows Neon to replace conventional backup strategies and simplifies database management by focusing on retention periods and branches, thus streamlining processes like point-in-time recovery without the complexity of managing multiple replicas or backup schedules.

Bunnyshell, a cloud-first Environments-as-a-Service platform, has integrated Neon Postgres to enhance the creation of on-demand and ephemeral environments for development, staging, and QA. This integration allows engineering teams to focus on core product development by eliminating the need to maintain local environments, ensuring production-like services and dependencies are available early in the development process. Neon's branching feature offers scalable and isolated production-like databases, which are crucial for short-lived environments, significantly improving developer productivity by reducing the time traditionally required for database seeding. The ability of Neon Postgres instances to automatically scale based on demand further supports efficient resource management. This strategic choice aims to streamline the feedback loop and increase developer efficiency, making Bunnyshell's platform more robust and adaptable for modern engineering needs.

The release of autoscaling in Feature-Preview allows databases to accommodate varying workloads by automatically increasing or decreasing CPU and memory allocation based on the current load. This feature provides several benefits, including accommodating varying and unexpected workloads, being highly scalable, saving time for manual management, scaling without restarting, simplified management, cost efficiency, and preventing overcommitting resources. The autoscaling system is implemented using a combination of Kubernetes, NeonVM, and cgroups to manage the allocation of resources to Postgres instances running in virtual machines. The system can handle live migrations of VMs to reduce load on nodes, and also includes a local file cache to speed up queries by utilizing additional memory when available.

Neon has introduced an autoscaling feature in a Feature-Preview that automatically adjusts the CPU and memory allocation of serverless Postgres databases based on workload demand, enhancing efficiency and adaptability. This autoscaling minimizes manual intervention and optimizes resource usage, ensuring cost efficiency and simplifying database management. By decoupling storage from compute endpoints, each Postgres instance operates within its own VM in a Kubernetes cluster, with the autoscaler-agent managing scaling decisions based on metrics. The system addresses challenges such as resource overcommitment and sudden memory spikes by coordinating with the Kubernetes scheduler and utilizing NeonVM for VM management. Additionally, enhancements like live migration and a local file cache improve performance and responsiveness, allowing for seamless scaling and efficient handling of queries.

The Neon serverless Postgres driver has reduced the latency of connections by cutting network round-trips in half. It achieves this by redirecting the PostgreSQL TCP wire protocol over WebSockets, eliminating the need for a separate WebSocket-to-TCP proxy. The driver uses TLS 1.3 to establish connections and eliminates unnecessary round-trips such as the `SSLRequest` message exchange. Additionally, it has improved Postgres authentication speed by using simple password auth instead of SCRAM-SHA-256, and pipelining Postgres interactions to reduce round-trips. With these improvements, the driver now achieves a total of four network round-trips, which is the lowest possible with current technologies. The team hopes to further optimize performance with WebSockets over QUIC in HTTP/3 and pipelining independent queries in Postgres.

Neon's serverless driver optimizes PostgreSQL connections by redirecting the TCP wire protocol over WebSockets, which is particularly advantageous for serverless environments that lack state persistence. This approach reduces latency by minimizing network round-trips, from an initial nine down to four, through various strategies such as upgrading to TLS 1.3, eliminating unnecessary `SSLRequest` round-trips, and employing Postgres pipelining. Additionally, the driver replaces the SCRAM-SHA-256 authentication method with simple password authentication to further decrease latency and CPU usage, making it suitable for serverless platforms like Cloudflare Workers. The integration of TCP_NODELAY and direct WebSocket connections to the main Postgres proxy also contribute to the reduction of latency. Future improvements may involve leveraging WebSockets over QUIC in HTTP/3, potentially decreasing round-trips even further.

The release of pg_tiktoken, a Postgres extension for fast BPE tokenization, has been announced. This new extension provides efficient text data analysis and processing within Postgres databases using the Byte Pair Encoding (BPE) algorithm. It is a wrapper around OpenAI's tokenizer, known for its speed and performance in natural language processing tasks. The tiktoken_encode function allows users to tokenize text inputs, while the tiktoken_count function returns the number of tokens in a text. Supported models include cl100k_base, p50k_base, p50k_edit, and r50k_base (or gpt2). The extension is optimized for speed and efficiency and supports various text inputs, including multiple languages and special characters.

The pg_tiktoken extension, released on the Neon platform, introduces swift and efficient tokenization within Postgres databases using the Byte Pair Encoding (BPE) algorithm. This extension facilitates the analysis and processing of text data by integrating OpenAI’s tokenizer, known for its high performance in natural language processing tasks. Key functions like tiktoken_encode and tiktoken_count allow users to tokenize text inputs and determine token counts, aiding in managing text length constraints, such as those in OpenAI's language models. By supporting various text inputs, including multiple languages and special characters, pg_tiktoken optimizes database queries and enables efficient handling of conversational data within token limits, particularly useful for applications involving language model interactions like ChatGPT.

The article provides a step-by-step guide on building an AI-powered ChatBot using Vercel, OpenAI, and Postgres. It covers the basics of word embeddings, their importance in natural language processing, and how to use OpenAI's API to create embeddings, store them in Postgres using the pgvector extension, and generate answers using OpenAI's completion method. The article also discusses fine-tuning the model using user feedback and gathering data using the OpenAI CLI. It highlights the importance of optimizing performance and user experience while reducing costs.

The second edition of Neon Developer Days is scheduled for March 29th, 2023, at 9:00 AM PT. This event will feature technical sessions and product announcements led by the company's engineers and team members. Attendees can expect to learn about various topics related to serverless Postgres, including building an AI-powered chatbot, scaling serverless Postgres, optimizing performance, understanding storage costs, and more. Registration is open now for those interested in attending this informative event.

The article presents a comprehensive guide on building an AI-powered chatbot using Vercel, OpenAI, and Postgres, with a focus on the implementation of word embeddings to tailor a ChatGPT-like application. It explains the concept and utility of word embeddings, which represent words as vectors and enhance the efficiency of language models by using low-dimensional feature vectors. The guide illustrates how to store these embeddings in Postgres using the pgvector extension and perform similarity analysis, leading to the creation of a chatbot application, Ask Neon, that can answer questions related to Postgres and Neon. The article details the steps to build the backend of the application using Vercel Edge Functions and the OpenAI API, from generating question embeddings to retrieving context and creating text completions. It emphasizes the potential for fine-tuning models by collecting user feedback to improve response quality and details the process for implementing such improvements. The guide further discusses strategies to enhance performance, reduce costs, and streamline user experience, culminating in a discussion on fine-tuning models based on user feedback using OpenAI's CLI tools.

Neon Developer Days is set to return for its second edition on March 29th, 2023, providing a full day of technical sessions and product updates led by the company's engineers. Attendees can expect in-depth discussions on topics such as the Neon storage engine, building AI-powered chatbots with Vercel, OpenAI, and Postgres, as well as insights into scaling and optimizing performance in serverless Postgres databases. The event aims to offer valuable knowledge on serverless Postgres, including managing storage costs and improving connection speeds, with additional focus on developer workflows using Vercel and GitHub Actions. Neon encourages participants to register early to secure their unique tickets and stay informed about further updates.