June 2024 Summaries

The text discusses a multi-expert problem involving green energy companies, where data is fetched from Wiki API, calendar, and a database. It then computes the largest increase in share prices using a calculator and formats the answer with a language model. The process involves challenges such as training discrete experts, interfacing them with neural networks, routing among modules, etc. Further discussion includes advantages of Jurassic-X, like reading and updating databases in free language, enabling joining multiple databases, and updating data using natural language commands.

AI21 Labs Co-CEO Yoav Shoham discusses how they built Jamba-Instruct, a foundation model with a context window of 256K tokens, to close the gap between claimed and effective context window length. The model is designed to efficiently serve long context workflows and offers a longer context than most competing models. Key questions addressed include whether having a long context window means the model does something useful with it, if long context models can be served with acceptable latency and unit economics, and if long context matters as much in RAGish days.

The text discusses a multi-expert problem involving green energy companies, where data is fetched from Wiki API, last month dates are extracted from the calendar, and share prices are retrieved from a database. The largest increase in share prices is then computed by a calculator before being formatted by a language model. It also mentions challenges in implementing multi-expert reinforcement learning (MRKL) systems, such as training discrete experts, interfacing with neural networks, and routing among modules. Additionally, it highlights the advantages of Jurassic-X, including its ability to read and update databases using free language, enabling users to interact with their data in natural language.

Transformer models have been successful in various AI applications but struggle with long texts due to memory usage and processing speed limitations. This issue affects real-world applications like report analysis, contract review, and chat transcripts. Jamba, developed by AI21 Labs, offers a solution by using a sequential approach inspired by human comprehension and combining Transformer layers with Mamba layers and Mixture-of-Experts modules. Jamba's hybrid architecture allows for high throughput and reduced memory footprint when processing long contexts, making it more efficient and cost-effective than traditional dense models.

AI21 Labs' Contextual Answers, a Task-Specific Model optimized for question answering tasks, outperforms leading Foundation Models such as Claude 3 Sonnet and Haiku, GPT-4 Turbo, GPT 3.5 Turbo, and Mixtral 8x7B in terms of output accuracy, context integrity, and answer relevance. These metrics are crucial for determining the reliability of a language model's question answering capabilities. Contextual Answers demonstrates its ability to minimize hallucinations by correctly identifying when an answer cannot be found within the given text, providing relevant answers without unnecessary embellishments, and maintaining context integrity.

AI21 Labs, a leading language model company, has announced its partnership with Snowflake Cortex to make AI21's groundbreaking large language models more widely accessible through Snowflake Cortex's fully-managed service. The integration of AI21's state-of-the-art Foundation Model, Jamba-Instruct, into the Snowflake Data Cloud will enable customers to analyze text data and build AI applications using a range of industry-leading LLMs, AI models, and vector search capabilities. This partnership aims to democratize access to cutting-edge technology, making powerful GenAI use cases securely within reach for all Snowflake customers.