Fast embeddings on GPUs

Perplexity describes a custom embedding-model serving stack designed to support both high-throughput indexing and low-latency online search, using its own models such as pplx-embed to improve search relevance, cost, and response times. The system combines Ivy, a Rust HTTP gateway that handles request processing, tokenization, batching, and load balancing; Tulip, a Rust-based gRPC inference server responsible for scheduling batches; and ROSE, a Python-based engine that executes model inference on accelerators while reusing much of the company’s LLM-serving infrastructure. The architecture treats bulk embedding workloads similarly to LLM prefill and short online queries similarly to decode, allowing shared kernels and components across model types. Key optimizations include whole-model CUDA graphs to reduce CPU kernel-launch overhead, lazy graph capture to limit startup costs, and LazyTensor-based asynchronous result handling to overlap CPU preparation with GPU computation. Perplexity also uses request splitting, in-house tokenization, ragged attention without KV caches for embedding models, and multiple attention-kernel backends selected by model shape and sequence length. Benchmark comparisons with vLLM are presented for latency, scoring, throughput, and concurrent-request performance, and the company concludes that ownership of the full Ivy–Tulip–ROSE stack provides greater efficiency and flexibility while retaining reusable infrastructure for future embedding and LLM improvements.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.