July 2024 Summaries

Arcee AI has introduced Llama-3.1 training and merging capabilities within its cloud services and tools like Arcee Cloud, Enterprise VPC, and mergekit, providing a robust solution for creating domain-adapted Small Language Models (SLMs) with extended 128K context, addressing the need for longer context in language models. Llama-3.1 allows continuous pre-training on proprietary text, maintaining knowledge transfer without loss even when using smaller context windows, facilitated by Arcee Spectrum's scanning and integration into continuous pre-training routines. Users can also fine-tune Llama-3.1 for specific tasks, enhancing its adaptability for various applications, while Arcee AI encourages merging of the model post-training to optimize utility. As the distinction between open-source and closed-source AI narrows, Arcee AI is dedicated to supporting the adaptation of powerful open-source models like Llama-3.1 for specialized domains, emphasizing the strategic advantage of this approach and inviting users to explore these capabilities through Arcee Cloud.

Arcee AI, a platform specializing in training Small Language Models (SLMs), collaborates closely with MongoDB to help customers in sectors like finance and insurance build domain-specific language models using their existing data infrastructure. MongoDB, known for its developer-centric database platform, integrates with Arcee AI to allow users to convert JSON data into parquet files, facilitating efficient model training and deployment. This partnership leverages MongoDB's robust database capabilities to support the growing needs of generative AI applications worldwide, with Arcee AI simplifying the process for users to develop custom language models quickly. MongoDB's commitment to innovation in generative AI is further exemplified by its new program for enterprises to build modern applications with advanced capabilities, emphasizing the importance of partnerships in advancing AI-driven solutions.

Two significant datasets have been released to the public to support AI research and development: the Agent Data and the Tome Dataset. The Agent Data, crucial for training the Arcee-Agent, includes Salesforce-xlam, agent-flan, a custom version of Glaive-FC2 with 20,000 extended samples, and Magpie-Pro, which collectively enable sequential tool use within responses and help maintain general capabilities while preventing catastrophic forgetting. The Tome Dataset contains 1.75 million samples of highly filtered data used for training Arcee-Spark and Arcee-Nova, aimed at developing generalist AI assistance. These releases are part of a commitment to transparency and collaborative advancement in AI research, inviting researchers and developers to explore and utilize the datasets responsibly, potentially leading to innovative applications and insights.

Arcee AI emerged as a response to the challenges enterprises face with generative AI, particularly regarding performance and security concerns associated with both closed-source and open-source models. Founded by engineers Mark McQuade, Brian Benedict, and Jacob Salowetz, the company quickly gained traction after its launch in Miami, becoming a go-to solution for secure and efficient GenAI models. Arcee's innovative platform, which operates in a virtual private cloud, allows companies to build and train models securely using their proprietary data, making it particularly appealing to highly regulated industries like legal, healthcare, insurance, and financial services. The company has successfully raised $5.5 million in seed funding and an additional $24 million in a Series A round, aiming to introduce small language models (SLMs) to the masses and launch Arcee Cloud, a hosted SaaS version of their platform. Arcee's focus on efficiency and specialization enables businesses to deploy models faster, with the ability to scale across multiple use cases, while technological advancements like Model Merging and Spectrum optimize performance and reduce costs. As Arcee AI continues to attract significant venture capital interest, it positions itself as a leader in the GenAI space, committed to democratizing AI access and facilitating continuous experimentation and optimization for diverse applications.

Arcee AI has introduced Arcee-Nova, a highly advanced open-source model that ranks as the top-performing model on the OpenLLM Leaderboard 2.0, reaching performance levels close to GPT-4 as of May 2023. Arcee-Nova is a fusion of Qwen2-72B-Instruct and a custom-tuned model, enhanced with Reinforcement Learning from Human Feedback (RLHF), and trained on a diverse dataset. The model demonstrates strong capabilities in areas such as reasoning, creative writing, coding, and general language understanding, making it suitable for numerous applications including customer service, content creation, software development, data analysis, and legal compliance. The development acknowledges contributions from the open-source community and invites further exploration and development by researchers and businesses.

Arcee-Scribe is a newly released language model designed to assist with creative writing tasks by combining the chat capabilities of Internlm2.5 with a diverse dataset that includes general and writing-specific data. Developed during the creation of Arcee-Spark V2, Arcee-Scribe benefits significantly from the post-training efforts of the Internlm team. It serves as a versatile writing assistant, adept at handling various tasks such as challenging dialogue, integrating technical concepts, and crafting complex narrative structures. Additionally, Arcee-Scribe is equipped for business applications, including content creation for blogs and social media, generating product descriptions, drafting customer communication, and creating training materials. The model aims to support businesses in their content development and foster idea generation in writing and communication, with its potential use in both creative and professional settings being eagerly anticipated. Notably, the blog post announcing Arcee-Scribe was largely written by the model itself.

Arcee AI has made significant strides this week with a $24 million Series A funding round led by Emergence Capital, alongside other investors, to advance the reach of its Small Language Models (SLMs) powered by Model Merging and Spectrum. The company introduced Julien Simon, a respected figure in AI, as their new Chief Evangelist, highlighting a shift in the generative AI market towards SLMs. Additionally, Arcee AI launched Arcee Cloud, a cloud-based platform for training, merging, and deploying SLMs, aiming to make domain-specific language models more accessible. The company is encouraging feedback from organizations to optimize their end-to-end platform tailored to various industry needs.

Arcee AI is gaining traction in the AI landscape by securing a $24 million Series A funding round, which follows a $5.5 million seed round, and launching Arcee Cloud, a hosted SaaS platform complementing its existing Arcee Enterprise deployment option. The company is focusing on small language models (SLMs) for domain-specific applications, such as HR, tax assistance, and medical inquiries, which offer benefits in cost, energy efficiency, and deployment flexibility over larger models. Arcee's innovative techniques, Model Merging and Spectrum, enhance model efficiency by combining attributes from multiple models without increasing size and optimizing training processes, reducing time and resources needed. This positions Arcee to capitalize on the trend away from large generic models, emphasizing tailored, agile AI development, potentially democratizing AI access across various industries. The company adopts an annual contract model for its SaaS and in-VPC offerings, aiming to provide predictable revenue while expanding customer capabilities. With a vision of making AI more agile and customizable, Arcee AI could reshape AI strategies, allowing businesses to explore multiple applications more feasibly.

Arcee AI emphasizes the importance of data quality and quantity in training artificial intelligence models, particularly Small Language Models (SLMs) and Large Language Models (LLMs). Ensuring a large, diverse, and high-quality dataset is crucial for developing effective language models, as it enables them to generalize across a variety of topics and tasks. The guide outlines several key considerations, including the utility of synthetic data to overcome data scarcity, the need for data filtering to remove undesirable content, and the significance of deduplication to enhance model robustness. Additionally, the text discusses the impact of temporal, content, and language shifts on model performance, highlighting the necessity of ongoing monitoring post-deployment to maintain accuracy. Arcee AI offers assistance to organizations in preparing their data for training and deploying custom SLMs on their platform, underscoring the critical nature of these preparatory steps in achieving optimal model performance.

Arcee AI has developed Spectrum, an innovative training methodology for large language models (LLMs) that enhances efficiency by selectively training layers based on their signal-to-noise ratio (SNR). Spectrum identifies layers with high SNR, which are crucial for performance, and focuses training efforts on them while freezing low SNR layers. This approach reduces training time, enhances memory efficiency, and minimizes catastrophic forgetting, enabling the training of large models like Qwen2-72B and Llama-3-70B on a single H100 node without sacrificing performance. Spectrum has improved the speed and quality of Arcee AI's model training processes, resulting in a 35% average reduction in training time and 36% reduction in memory usage, while maintaining or improving performance metrics. The methodology has been validated against other techniques like QLoRA and full fine-tuning, confirming its effectiveness in Continual Pre-Training (CPT) and Supervised Fine-Tuning phases, ensuring Arcee AI remains at the forefront of LLM training innovations.

Arcee Agent is a newly released 7 billion parameter language model optimized for function calling and tool usage, derived from Qwen2-7B. It offers high performance akin to larger models but with greater efficiency and speed, excelling in advanced function calling, multi-format support, and dual-mode functionality as both a tool router and standalone chat agent. Designed for real-time applications, Arcee Agent is proficient in API integration, database operations, and code generation, making it valuable across industries such as customer support, sales, marketing, finance, healthcare, and more. Trained using Spectrum with compute from CrusoeAI, the model is ideal for businesses seeking sophisticated AI solutions without the heavy computational demands of larger models, though it may have limited capabilities outside its specialized domains. Users are advised to validate outputs and use the model responsibly while exploring its potential in various applications.