August 2024 Summaries

In this tutorial, you will learn how to create a local semantic search engine using Pinecone and the Hugging Face Transformers library. This search engine is capable of searching through text data based on its meaning rather than just keywords. The main components of this search engine are: 1. A Pinecone index, which stores the embeddings of your documents. 2. A transformer model from the Hugging Face Transformers library, which generates these embeddings. 3. An inner product search function that retrieves the most relevant documents based on their similarity to a given query embedding. To create this search engine, you will need: 1. Node.js and npm installed on your machine. 2. A Pinecone account with an index created. 3. The Hugging Face Transformers library installed in your project. Here's a step-by-step guide to building the search engine: 1. Set up your project: - Create a new directory for your project and navigate into it. - Run `npm init` to create a package.json file, then install Pinecone and the Hugging Face Transformers library by running `npm install pinecone-client @huggingface/transformers`. 2. Create an environment variable for your Pinecone API key: - Add a new entry to your .env file with the key `PINECONE_API_KEY` and set its value to your actual Pinecone API key. 3. Implement the search engine: - In your main JavaScript file (e.g., index.js), import the required modules and initialize a new Pinecone client with your API key from the .env file. - Define a function to insert documents into the Pinecone index. This function should take an array of document objects, where each object has two properties: `id` (a unique identifier for the document) and `text` (the content of the document). The function should then use the transformer model to generate embeddings for each document, insert these embeddings into the Pinecone index along with their corresponding metadata, and return an array containing the IDs of all inserted documents. - Define a function to search through the documents in the Pinecone index based on a given query text. This function should use the transformer model to generate an embedding for the query text, then perform an inner product search using this query embedding and return the top 10 most relevant documents along with their similarity scores. - Finally, test your search engine by inserting some sample documents into the Pinecone index and performing a few searches. That's it! You now have a fully functional local semantic search engine that can efficiently search through text data based on its meaning. This search engine is perfect for applications like chatbots, question answering systems, or any other use case where you need to find relevant information quickly and accurately.

Vercel has added official First-Party Integrations, including Supabase, making it easier to launch Postgres databases from the Vercel dashboard with full support for Vercel Templates and integrated billing. This integration allows users to manage all their Supabase services directly from the Vercel dashboard, with credentials automatically injected into the Vercel environment. The combined use of Vercel and Supabase has been popular among scale-ups, YC companies, and large enterprises. Some key features include Pure, Dedicated Postgres, Extended, modular building blocks, Templates, Low latency & Read replicas, Integrated billing, and predictable pricing with spend caps for developers. The Supabase Starter on the Vercel Template marketplace provides a quick way to get started with this integration.

The Supabase Hackathon is set to take place from September 13th at 09:00 am PT to September 22nd at 11:59 pm PT, allowing participants 10 days to build a new Open Source project using Supabase. Participants can enter as individuals or teams of up to four people and are encouraged to use the hackathon as a launch pad for their new Startup ideas, side projects, or indie hacks. There will be five categories with prizes for Best Overall Project, Best Use of AI, Most Fun/Best Easter Egg, Most Technically Impressive, and Most Visually Pleasing. The Supabase Team will judge the winners based on creativity, functionality, visual appeal, technical impressiveness, use of Supabase features, and fun factor. All entries must be Open Source and use Supabase in some capacity.

Supabase has added support for Mozilla Llamafile, an Inference Server that can be used with Edge Functions. Llamafile allows users to distribute and run Language Models with a single file, running locally on most computers without installation. It also provides a local web UI chat server and an OpenAI API compatible server now integrated with Supabase Edge Functions. Users can find examples of how to use Llamafile with functions-js on GitHub. Instructions for getting started are provided, including setting up a new Supabase project locally and calling Llamafile with functions-js or the OpenAI Deno SDK. Deploying a Llamafile and Supabase Edge Functions is also covered in the text.

Launch Week brought several updates to Supabase, including open-sourcing Snaplet, adding new tools for data copying, seeding databases, and taking snapshots. Other updates include pg_replicate for transferring Postgres data to other systems, a CLI utility for migrating vector database data to Supabase or any Postgres instance, an official VS Code extension, Python library support, Log Drains for exporting logs, Realtime authorization improvements, third-party Auth provider support, phone-based multi-factor authentication, new Auth hooks, Wasm Foreign Data Wrapper support, and the release of postgres.new, a browser-based Postgres sandbox with AI assistance. Additionally, there is now an entire book available on Supabase.

PostgREST 12.2 introduces observability and API improvements, including Prometheus-compatible metrics for the schema cache and connection pool, hoisted function settings allowing custom timeouts per function, and a max-affected preference to limit affected rows up to a custom value. These enhancements improve troubleshooting and provide greater flexibility in managing SQL statements and mutations. The latest version is available on the Supabase platform for new projects or as an upgrade for existing ones.

The Supabase community has seen a surge in demand for client libraries and framework-specific SDKs, with dozens of open source libraries currently maintained. As certain libraries achieve broad adoption, they may be considered for official support. Recently, the Python Client Libraries have been officially supported on the Supabase platform due to their significant growth in usage, particularly within the AI and ML community. The latest updates include enabling HTTP2 by default, automatically following redirects, keeping connections alive, supporting Edge Functions regions, upgrading Realtime V2, improving Auth features, enhancing Postgrest quoting/escaping in queries, allowing unverified SSL, closing socket in Realtime, setting timeouts for Edge Functions, and creating a new tool to migrate vector data.

Foreign Data Wrappers (FDWs) in Postgres allow interaction with externally hosted data by creating foreign tables. Today, support for WebAssembly (Wasm) wrappers has been released, enabling users to create Wasm wrappers to external services and run them directly from platforms like GitHub. This feature enhances security, simplifies development, distribution, and improves performance of FDWs. The architecture includes a Supabase Wrappers Extension that runs within Postgres and isolated, dynamically-loaded Wasm packages (guests) for data fetching and processing. Currently, the Wasm FDW supports HTTP(s)-based JSON API sources but not TCP/IP-based DBMS or local files. Developers can build their own Wasm FDWs to integrate diverse data sources into Postgres. The feature is available on the Supabase platform with built-in wrappers for Snowflake and Paddle.

At Supabase, we're constantly striving to provide the tools developers need to build secure, reliable applications. Our latest update focuses on an area that's critical to both security and reliability: Platform Access Control. We're excited to announce the rollout of our new granular access control features which allows giving users access to specific projects instead of the entire organization. This feature fills a gap in managing platform-level access by offering Role-Based Access Control (RBAC) to the Supabase platform and management APIs. With this update, developers can now manage permissions at both the organization and project levels, assigning roles such as Owner, Administrator, Developer, and Read-Only to team members. By doing so, teams can reduce the risk of accidental changes, streamline workflows, and maintain a high level of security across their projects. The new features are designed to unlock the full potential of teams by providing granular control over access, making it easier than ever to ensure that every team member has the right level of access.

vec2pg is a CLI utility that facilitates the migration of data from various vector databases into Postgres with associated ids and metadata. The tool supports migrating from Pinecone and Qdrant, and it allows users to manipulate the data using SQL to transform it into their preferred schema. When migrating, users should increase their Supabase project's disk size to accommodate the vectors. The performance of `vec2pg` is dependent on several factors, including network speed, vector dimensionality, and metadata size. Pinecone and Qdrant collections can be migrated at rates of 700-1100 records per second and 900-2500 records per second, respectively. Postgres/pgvector offers several benefits, including performance, scalability, security, backups, row-level security, and compatibility with a wide ecosystem of tools. The tool is designed to make it easy for users to efficiently copy their data from various vector databases into Postgres, allowing them to manipulate the data using SQL and transform it into their preferred schema.

David Lorenz's new guide to learning Supabase is a comprehensive resource that aims to help users master the full potential of the platform. As a seasoned software architect with over two decades of experience, David has created a multi-part book covering topics such as creating foundations, adding multi-tenancy and learning Row-Level Security (RLS), managing tickets and interactions, and diving deeper into security and advanced features. The guide is designed to be thorough and detailed, with the goal of providing users with exceptional knowledge that sets them apart from others in the Supabase community. David has also set up a Discord channel for support and feedback, and invites readers to share their thoughts on the book and tag him on social media.

In version 1.5.7 of pg_graphql, several quality of life improvements have been made. These include support for pagination via First/Offset, schema-based multi-tenancy, and filtering on array typed columns with contains, containedBy, and overlaps operators. The introduction of offset-based pagination allows users to skip a certain number of records before returning the results. Multi-tenancy performance has been improved by caching only schemas that tenants have usage permission for, reducing query time in multi-tenant environments. Additionally, filtering array column types is now supported for scalar array fields like text[] or int[]. Future releases will include support for insert on conflict/upsert and nested inserts.

Supabase has released Log Drains for Team and Enterprise users, allowing developers to export logs generated by their Supabase products to external destinations such as Datadog or custom HTTP endpoints. This feature supports all logs from Database, Storage, Realtime, and Auth products. Beyond providing a single pane of glass inside existing logging and monitoring systems, Log Drains can be used for additional alerting and observability pipelines. Configuring Log Drains is done in the project settings with initial supported destinations including Datadog Logs and HTTP Endpoint. The Supabase Analytics server, built into Logflare, has undergone a multi-year effort to allow multiple backends to be used with the server. Log Drains are available for self-hosting and local development through Studio under Project Settings > Log Drains. Pricing starts at $60 per month per project, with additional fees based on log events processing and egress. Future developments include support for Syslog and Loki destinations, log sampling, draining specific product logs, and sharing Log Drains between multiple projects.

Supabase Auth has introduced three new features: support for third-party authentication providers, phone-based multi-factor authentication (SMS and WhatsApp), and new Auth Hooks for SMS and email. The platform now supports first-class integration with popular third-party auth products like Auth0, AWS Cognito, and Firebase Auth. Additionally, developers can use Supabase's native auth offering alongside their preferred third-party authentication provider to achieve a disruption-free migration. Furthermore, phone-based MFA is introduced for non-techies who find app authenticators difficult to adopt. Lastly, new Auth Hooks are added for customizing email and SMS sending processes.

Despite the shutdown of startup Snaplet, its founder Peter Pistorius has decided to open source three main tools under the MIT license: Copycat, Snaplet Seed, and Snapshot. These tools are designed for generating fake data, creating realistic synthetic data based on a database schema, and capturing, transforming, and restoring snapshots of databases. The Snaplet team has joined Supabase to help with the migration of these projects to open source. Meanwhile, Peter Pistorius is back at RedwoodJS, working on React Server Components.

Authorization for Realtime's Broadcast and Presence allows users to control access to channels using Row Level Security policies, enabling fine-grained authorization for sending and receiving messages, presence updates, and other operations within these channels. To implement this feature, users create RLS policies that define the permissions for a specific channel or topic, specifying whether a user can read, write, or perform other actions on the channel. Realtime checks these policies against the user's claims when they connect to a channel, ensuring that only authorized users can access and modify data within those channels. The approach is performance-oriented, caching policies in memory to minimize latency and maximizing throughput. This feature is currently available in Public Beta, with ongoing improvements and feedback encouraged from developers.

GitHub has released a new extension for VS Code to enhance development with Supabase and VS Code. The extension includes features such as GitHub Copilot Chat Participant, guided database migrations, inspecting tables & views, listing database migrations, inspecting database functions, and listing storage buckets. It requires the Supabase CLI installed and running locally. Future updates will integrate the Supabase Management API for seamless connection to hosted Supabase projects. The Supabase stack is fully open source, and contributions from users are highly valued in shaping its future.

Postgres.new` is an in-browser Postgres sandbox with AI assistance that allows users to spin up unlimited number of databases and run them locally without the need for remote containers or WebSocket proxies. The application uses a WASM version of Postgres called PGlite, which can run directly in the browser, and pairs it with a large language model (LLM) to provide features such as drag-and-drop CSV imports, generate charts, build database diagrams, and more. The AI assistant is fully autonomous over the database, allowing for faster execution of queries without delay. The application also supports semantic search and RAG using Postgres extensions like pgvector and live. With `postgres.new`, users can deploy their databases to S3, share them with others, and access them from anywhere on the internet, making it an ideal solution for developers who want to test and experiment with different database configurations without incurring high costs.