Replit trains its Large Language Models (LLMs) using a combination of Databricks, Hugging Face, and MosaicML. The company aims to reduce dependency on external providers, increase customization, and improve cost efficiency by training its own models from scratch. This approach allows Replit to tailor its models to specific needs, including platform-specific capabilities and terminology. The data pipelines used are robust and highly optimized, with tools like Databricks providing scalable and tractable analytics. MosaicML is used for model training, offering benefits such as multiple cloud providers, well-tuned configurations, and managed infrastructure. The models are deployed into production using NVIDIA's FasterTransformer and Triton Server, which accelerates inference and allows for ultra-fast distributed inference of large models. Replit continues to monitor model performance and usage metrics, gathering feedback and iterating rapidly to improve its LLMs.