Prem Benchmarks is an open-source project designed to evaluate the performance of various large language model (LLM) inference engines, such as vLLM, TensorRT LLM, and HuggingFace Transformers, across different precisions including float32, float16, int4, and int8. The project aims to provide the open-source LLM community and enterprises with insights into LLM inference metrics by comparing distinct open-source implementations. The benchmarks focus on important decision-making factors like latency, memory usage, and cost-effectiveness, which are crucial for determining the best-suited inference engine for specific requirements. This initiative also emphasizes the importance of reproducibility, transparency, and the trade-off between optimization and quality in LLM deployments. The benchmarking process involves a structured approach using Python scripts for installation and benchmarking, with results categorized by performance and quality comparisons. The project currently benchmarks models like LLama 2 7B chat and Mistral v0.1 Instruct, with Nvidia's TensorRT LLM identified as a leading performer in terms of speed and quality consistency, despite its higher GPU memory usage. The initiative seeks to continuously update and expand its benchmarks to include newer engines and encourage community contributions to enhance its comprehensive repository.