October 2023 Summaries

Perplexity announced pplx-api, a now-deprecated public-beta REST API that provided access to open-source language models including Mistral 7B, Llama 2 variants, and Code Llama, with OpenAI-client compatibility intended to simplify adoption. The service emphasized managed deployment, fast inference, and scalable infrastructure built on AWS A100 GPUs, NVIDIA TensorRT-LLM, and Kubernetes, claiming lower latency and higher throughput than several competing inference approaches in internal benchmarks. Perplexity reported that using the API for a production feature reduced annual costs by roughly $620,000 while sustaining more than one million daily requests and nearly one billion tokens per day without statistically significant quality changes in A/B testing. The announcement described API authentication and chat-completion usage, outlined plans for additional models, embeddings, pricing, and retrieval-augmented generation features, and stated that API data would be deleted after 30 days, would not be used for training, and could be excluded from retention through account settings.