August 2023 Summaries

The document details the use of Deep Infra's platform for running OpenAI API-compatible models, emphasizing its cost-effective pricing of $1 per million tokens and robust GPU infrastructure designed for scalable AI hosting with enterprise-grade uptime. It outlines the process of setting up a virtual environment, installing the OpenAI Python client, and configuring the API key, base, and model parameters for chat completion using various models, such as meta-llama and CodeLlama. Both streaming and batch modes are supported, and users can switch from existing OpenAI integrations by adjusting key parameters. Additionally, the platform features the latest AI models and provides comprehensive documentation for more detailed guidance.

Published by Nikola Borisov on August 9, 2023, the text discusses the decision by a team to fork the text generation inference project initially by Hugging Face due to a change in its licensing from Apache 2.0 to a more restrictive version with v1.0.0. This fork aims to maintain the project under the original Apache 2.0 license, keeping it open source and encouraging community contributions. The article highlights a growing trend where popular open-source projects alter their licenses after gaining traction, posing challenges for developers who have integrated these projects into their applications. The author expresses concerns over mid-flight license changes as they can disrupt users, using examples like MongoDB, Grafana, and ElasticSearch, and speculates on the impact if similar changes were made to projects like PyTorch or Transformers. The team offers support via their Discord server and provides a link to the forked project's code, hoping to foster a community dedicated to truly open-source development.

Llama 2 models, released by Meta AI and available for commercial use, represent the latest advancements in open-source language models and can be utilized through DeepInfra to build AI applications cost-effectively. The models come in different sizes, including llama-2-7b, llama-2-13b, and llama-2-70b-chat, each varying in speed, cost, and accuracy to suit different application needs. Users can access these models by creating an account on DeepInfra, obtaining an API key, and following detailed API documentation to make inference requests via simple POST requests. DeepInfra offers a fully managed GPU infrastructure that ensures enterprise-grade uptime at competitive rates, providing a cost-effective alternative to OpenAI's API for running AI models at scale.