What We Learned Building O11y GPT: Part I

Observe uses OpenAI's large language models, specifically GPT-3.5, to enhance its O11y GPT Help bot, which assists users by providing succinct help and instructions based on their queries. The challenge lies in the model's token limitation of 4097, which restricts the amount of context and instructions that can be processed, especially given Observe's extensive 200,000-word documentation. To address this, Observe employs machine learning embeddings to efficiently map and retrieve relevant information from their vast documentation. They use the Ada-002 model to generate a 1536-dimensional vector for document segments, storing these in a FAISS index, which allows them to quickly identify and use the most pertinent document sections when answering user queries. Despite the model's wide-ranging pre-trained knowledge, it lacks specific updates on Observe's internal workings, leading to potential misinterpretations, especially where specialized syntax or rules are involved. To mitigate this, Observe simplifies its documentation into smaller sections, generates embeddings, and processes these efficiently, although unique challenges like syntax discrepancies still persist.