September 2026 Summaries

Neki addresses hot shards by allowing teams to adapt sharding strategies without downtime when uneven tenant growth, large datasets, or cross-tenant query requirements make an original shard key ineffective. Drawing on Slack’s Vitess-based evolution from sharding messages by workspace to sharding them by channel, it argues that tables should use shard keys aligned with their access patterns rather than relying on a single key for all data. Neki uses a declarative data topology to specify table shard keys and key ranges, making placement predictable and enabling resharding operations that copy existing data, stream ongoing writes, and switch traffic while applications remain available. Suggested responses to hot shards include temporarily scaling up an individual Postgres shard, isolating a large tenant into its own key range, or, as the more durable option, resharding selected high-growth tables by a more appropriate dimension. The recommended approach is to address the tables creating current bottlenecks incrementally, rather than waiting to redesign an entire database schema.

Choosing between aarch64 (ARM) and x86-64 for a PlanetScale Postgres or Neki database can significantly affect performance, cost, and future migration options despite identical advertised vCPU and memory allocations. ARM server processors provide one physical core per vCPU, making them well suited to concurrent workloads with many smaller queries, parallel workers, and autovacuum activity, while x86 instances often expose hyperthreads as vCPUs and may offer stronger single-core performance for CPU-bound queries. x86 can also benefit workloads using wide AVX2 or AVX-512 vector instructions, including some full-text search and pgvector operations, whereas current ARM cloud chips use narrower vectors. CPU architectures cannot be changed in place because PostgreSQL physical replication depends on byte-identical storage files whose interpretation can vary between architectures, so switching requires creating a new cluster and migrating data through logical replication. ARM is presented as a cost-effective default for typical read-and-write applications, while x86 merits consideration for workloads dominated by a small number of intensive queries; testing equivalent workloads on both architectures under expected concurrency is recommended before making a long-term choice.

Full-text search relies on inverted indexes rather than conventional b-tree indexes, which efficiently handle exact, range, and prefix matches but cannot search terms in the middle of text. An inverted index maps tokenized terms to sorted, compressed postings lists of document or row identifiers, optionally storing positional information for phrase and proximity searches and frequency statistics for ranked results using methods such as BM25. Index design involves trade-offs among size, speed, precision, and supported features, including tokenization, stemming, stop-word handling, bitmap encoding, and top-k query optimizations. To accommodate changing data, indexes are commonly divided into immutable segments plus mutable storage, with background merges consolidating segments and removing tombstoned documents created by deletes or updates. Implementing such an index in PostgreSQL adds database-specific requirements including row identifiers, transaction visibility, WAL logging, VACUUM compatibility, storage reuse, background maintenance, and query-planner integration with conventional SQL indexes; PlanetScale’s TIN extension is presented as an effort to address these challenges for PostgreSQL full-text search.

PlanetScale Postgres blocks planned database cutovers, such as resizes, configuration changes, and maintenance, when logical replication slots are not synchronized and ready on a promotion candidate, aiming to prevent downstream applications from losing change-data-capture events. While PostgreSQL WAL replication can ensure that replica data is current, its default promotion behavior does not require logical slots—WAL bookmarks used by analytics, search, and queue consumers—to be copied and usable on the new primary. PlanetScale’s Kubernetes operator detects slot and configuration problems, alerts users through email and dashboard notices, waits for required slots to become ready, and can eventually permit a blocked cutover after a grace period with a warning of potential data loss. To make slots promotion-ready, users must create or alter them with `failover = true`, register dependent slot names in PlanetScale, and enable `hot_standby_feedback` and `sync_replication_slots`; these settings preserve and synchronize slot state but can increase primary database bloat, particularly when replicas have long-running transactions.

Neki is a Kubernetes-first sharding and scaling system that uses unmodified PostgreSQL instances while presenting applications with a single PostgreSQL-compatible connection endpoint. Its architecture combines PostgresManager for instance lifecycle and replication setup, Sidecars for connection pooling, health reporting, and backend-session management, and shards consisting of a primary plus replicas for availability and read capacity. An authoritative shard maintains consistent custom type OIDs and schema information across independently managed shards, while Admin monitors health, replication lag, durability policies, and failovers or planned switchovers. The Kubernetes Operator deploys and replaces cluster components, and the Router authenticates clients, parses and plans SQL using the authoritative catalog, routes queries to appropriate shards, coordinates distributed joins and aggregates when necessary, and buffers retry-safe queries during transitions. An etcd-backed Data Topology provides a shared, dynamically updated map of shard groups, routing keys, and sharded tables. The Replicator supports online data movement for imports, table relocation, resharding, and schema changes through logical replication and controlled cutovers, enabling horizontal Postgres scaling with minimal disruption.

Lead is a PostgreSQL full-text search extension designed to provide TIN-compatible functionality for testing, CI, development, and staging environments where the faster TIN extension may not be available. It supports the same SQL statements, TINQL queries, tokenization behavior, BM25 scoring, hit highlighting, transactional writes, snapshot isolation, rollback handling, and restart durability as TIN, ensuring that search behavior remains consistent across environments. Unlike TIN, however, Lead intentionally performs every search as a full table scan and is not intended for production-scale workloads, with large datasets requiring minutes per query rather than milliseconds. The developers recommend using Lead with CI datasets of only a few megabytes while deploying TIN for production search applications, and invite user feedback on compatibility needs.

TIN, or Text INdex, is a generally available PostgreSQL extension for Postgres and Neki databases that provides full-text search through a simple index and query syntax while supporting Boolean, phrase, span, fuzzy, wildcard, regex, case- and accent-insensitive searches, BM25 ranking, exact counts, transactional visibility, updates, backups, replication, joins, and complex filters. Its developers position it as addressing capabilities and operational requirements not fully covered by PostgreSQL GIN, ParadeDB, or pg_textsearch, with examples spanning ranked product search, unranked legal discovery, and tag-counting applications. Benchmarks on a 150-million-document, 85 GB Stack Exchange corpus found that TIN built its index in about eight minutes using 32 GB RAM and, across tested workloads, delivered substantially higher throughput, lower p99 latency, and less data read per query than competing extensions; some alternatives could not complete certain workloads because of memory limitations or restricted query support. The extension’s design uses PostgreSQL’s native ctid tuple locations as document identifiers rather than maintaining separate sequential IDs, enabling compressed two-level bitmap postings, vectorized CPU operations, direct heap-order row access, and avoidance of identifier-mapping overhead. TIN also integrates MVCC visibility handling, VACUUM-driven liveness bitmaps, and background segment promotion and merging, with its ctid-based structure intended to reduce the rewriting, recompression, and write amplification commonly required when search-index segments are merged.

Neki, released in platform preview, was benchmarked at 118.5 million queries per second across 512 PostgreSQL-backed shards containing 1.22 PiB of data, substantially exceeding an initial goal of 1 million QPS. The test used isolated, single-shard primary-key point reads that fetched one row per query, without writes, joins, cross-shard queries, replicas, or failover testing. Results showed near-linear scaling from five shards at roughly 1 million QPS to 50 shards at 9.9 million and 512 shards at 118.5 million, with each shard reaching about 231,000 QPS. During a 16-minute sustained run, the system recorded p99 latency of 6.06 milliseconds at routers and 13.95 milliseconds at clients, approximately 67 errors per second, 15.8 million read IOPS, and more than 2 Tbps of network traffic.

Neki presents a sharded PostgreSQL architecture designed to make databases distributed across many servers appear to applications as a single Postgres instance, while handling authentication, wire-protocol compatibility, SQL parsing, distributed planning, failures, and connection pooling. Using a customer-and-orders join as an example, it traces how a router implements PostgreSQL SCRAM authentication and both simple and extended query protocols, parses SQL into an abstract syntax tree, consults cached schema and shard-topology metadata, and creates a distributed execution plan. When customers and orders are sharded independently by their IDs, related rows may reside on different shards, requiring scatter-gather requests and a router-side hash join that can spill to disk if memory is insufficient; individual shards still use PostgreSQL’s native planner to execute their assigned subqueries. Requests travel through sidecars that pool reusable Postgres connections and restore the correct session settings and role, while results are streamed back and, where possible, forwarded using their existing encoded values to reduce processing. For distributed aggregates such as AVG, the router rewrites calculations into shard-local sums and counts before combining them centrally. The discussion concludes that choosing compatible shard keys, such as sharding orders by customer_id alongside customers by id, colocates related data so PostgreSQL can perform joins locally, reducing router work and network requests.

PlanetScale has introduced Neki, a platform-preview service for sharded PostgreSQL designed from its experience operating large sharded MySQL deployments and addressing the limits of scaling a single Postgres machine. Neki distributes PostgreSQL databases across multiple machines while retaining standard Postgres clusters, SQL behavior, extensions, drivers, ORMs, and wire-protocol compatibility. Applications connect through scalable Neki routers that parse, plan, route, and combine distributed queries, while shard groups allow tables and workloads to be independently distributed and configured. Each shard includes a primary and at least two replicas across three availability zones, with sidecars providing managed connection pooling and a control plane coordinating monitoring, failovers, schema changes, upgrades, imports, and resharding. Database topology is configured through JSON definitions of shard keys, hashing, and shard groups, and online workflows aim to avoid maintenance windows. Neki can initially operate as an unsharded primary-and-replica cluster before customers reshard as demand grows, and it includes PlanetScale features such as Insights, schema recommendations, branching, and MCP. The company states that Neki remains under active development during the preview and should not yet be used for production workloads.

Neki is a Postgres-compatible routing layer for sharded databases that determines which shards should execute a query and coordinates results before each selected Postgres instance creates its own execution plan. A stateless fleet of routers sits between applications and database shards, handling client connections, session state, SQL parsing, query planning, and distributed processing without requiring a separate Postgres backend process per client connection or an additional PgBouncer layer. Routers send planned work over gRPC to sidecars beside each Postgres instance, which use pooled connections to forward requests and stream responses. Queries containing a shard key, such as a user ID, can be routed to one shard and use reusable parameterized plans, while queries without a shard-key predicate are scattered across all relevant shards and then gathered, limited, or otherwise combined by router operators. Neki’s EXPLAIN options reveal both its distributed routing plan and representative Postgres execution plans, helping users inspect routing choices, remote work, and runtime behavior. By independently scaling routers for connection handling and distributed query processing and shards for data storage and database capacity, Neki is designed to grow beyond the limits of a single Postgres instance.