March 2024 Summaries

The SLM Show is a biweekly live-streamed series focusing on Small Language Models (SLMs), hosted by Arcee CEO Mark McQuade and Mary MacCarthy. The show provides an informal exploration of SLM development, training, deployment, and maintenance, featuring discussions with top industry researchers and business leaders. Their first episode, which was well-received, introduced the concept of Model Merging, a technique that enhances machine learning models by combining their strengths, as explained by guests Charles Goddard, a Senior Research Engineer at Arcee and MergeKit founder, and machine learning scientist Maxime Labonne. Model Merging is noted for its cost-effectiveness and its ability to improve models without the need for extensive resources, allowing for decentralized experimentation. The hosts and guests also discussed the potential of Model Merging for avoiding catastrophic forgetting, the future of merging different model architectures, and its implications for AGI development. The show, accessible on platforms like LinkedIn, Twitter, and YouTube, invites ongoing viewer engagement and promises further exploration into the SLM domain.

Arcee's approach to domain adaptation for language models involves leveraging Continual Pre-Training (CPT) and Model Merging to efficiently and effectively fine-tune models for specialized fields such as the medical and patent domains. This case study highlights the challenges of traditional domain adaptation methods, notably catastrophic forgetting, and how Arcee's strategies mitigate these issues by integrating domain-specific knowledge while maintaining general capabilities. Through examples like PMC-LLaMA and ChipNeMo, the document illustrates how Arcee employs domain-adaptive CPT to significantly enhance model performance, achieving reductions in model size without sacrificing quality. Model Merging is also used to synthesize the strengths of various pre-trained models, further refining their adaptability and performance across specialized tasks. The research underscores the importance of high-quality CPT checkpoints and strategic merging methods, demonstrating the potential for cost-effective, high-performance models tailored to specific industry needs.

Small Language Models (SLMs) are increasingly being recognized as a transformative tool across various industries, offering domain-specific AI capabilities that enhance workflows and decision-making. In customer service, SLMs power chatbots that deliver efficient, personalized support, reducing costs and response times while improving accessibility and sentiment analysis. In healthcare, they assist in patient care by analyzing records and providing virtual support, thereby enhancing medical decision-making and administrative efficiency. The finance and banking sectors benefit from SLMs through automated customer service, fraud detection, and personalized financial advice, facilitating compliance and risk management. In the legal industry, SLMs streamline research and document review, offer predictive legal analytics, and improve access to legal assistance. Education is also being revolutionized by SLMs, which provide personalized learning experiences, automate grading, and enable gamified learning. SLMs address the limitations of Large Language Models (LLMs) by offering cost-effective, secure, and context-specific solutions for enterprise applications, as demonstrated by companies like Arcee, which lead in SLM adoption and integration.

Arcee's recent merger with the open-source repository, mergekit, founded by former NASA and Apple engineer Charles Goddard, highlights a commitment to advancing research in model merging, Large Language Models (LLMs), and Small Language Models (SLMs) while maintaining open-source accessibility. Goddard's initial interest in model merging was sparked by innovative techniques described in academic literature, but the lack of readily available code led him to create mergekit, which gained popularity on GitHub. Now part of Arcee as a Senior Research Engineer, Goddard emphasizes the importance of open-source tools for democratizing access to AI research and fostering community-driven experimentation, which he believes can produce more innovative results than closed, corporate environments. This collaboration aims to push the boundaries of human knowledge in AI, encouraging widespread participation and exploration in the field.

Arcee has rapidly advanced since its emergence from stealth mode, securing $5.5 million in seed funding from investors such as Wndrco, Long Journey Ventures, and Flybridge, and merging with the open-source model merging toolkit, mergekit. The co-founders of Arcee, Mark McQuade, Jacob Solawetz, and Brian Benedict, have prioritized addressing enterprise concerns about GenAI security and transparency by offering a robust end-to-end system for training and deploying GenAI models in a VPC, distinguishing themselves in the market. Their merger with mergekit, initiated by recognizing the transformative potential of model merging, aligns with Arcee's commitment to the open-source community and enhances their offerings in building small language models (SLMs). Charles Goddard, the creator of mergekit, joined Arcee as a Senior Research Engineer, and the company is focusing on promoting model merging through initiatives like March Merge Madness.

Arcee's Small Language Model (SLM) universe employs a technique called model merging, which enhances computational efficiency in Continual Pre-Training by training a smaller model and merging it with a larger one. This method, as detailed by Senior Research Engineer Charles Goddard, founder of mergekit, is considered superior to dataset blending techniques like DoReMi, particularly for domain-specific tasks. While traditional Continual Pre-Training can be compute-intensive and may cause degradation in a model's general intelligence, model merging allows Arcee to maintain the large language model's capabilities while also incorporating domain-specific reasoning. This approach effectively combines the strengths of both the base model and the specialized checkpoint, mitigating issues such as catastrophic forgetting without the extensive computational cost.

Arcee is focusing on model merging as a key component of its Small Language Model (SLM) system, which aims to provide cost-efficient and flexible solutions for business use cases by integrating smaller and larger models. The SLM system incorporates Continual Pre-Training, Supervised Fine-Tuning, and Retrieval Augmented Generation as its core pillars. By merging smaller, efficiently trained models with larger ones, Arcee seeks to address 99% of business needs without the extensive resource expenditure typically associated with training large models. CEO Mark McQuade emphasizes that this approach allows for high-performing models without the need for exhaustive parameter training, aligning with Arcee's commitment to becoming a leader in model merging.

Arcee has become a leader in model merging by partnering with mergekit, a toolkit developed by Charles Goddard, a former engineer from NASA and Apple. Goddard created mergekit to address the lack of code releases in cutting-edge research papers, providing a user-friendly and resource-efficient solution for the open-source community. Arcee quickly recognized the potential of mergekit, leading to a collaboration that brought both the toolkit to Arcee's GitHub repository and Goddard to their team as a Senior Research Engineer. Both Arcee and Goddard view model merging as a significant advancement in large language model (LLM) research and are committed to maintaining mergekit as open source. Model merging combines pre-trained checkpoints of language models to harness the strengths of multiple models, offering a valuable way to extend the utility of costly models even after newer versions are released. This technique is not limited to language models but is applicable across various domains such as computer vision and natural language processing, emphasizing the continued relevance and potential of pre-trained models.