June 2024 Summaries

Cursor, an AI-native integrated development environment (IDE), is revolutionizing code generation and editing by leveraging Fireworks' AI inference stack to achieve rapid token processing through a newly introduced Speculative Decoding API. This innovative API allows for parallel token generation, significantly enhancing the speed and efficiency of code editing tasks, a notable improvement over previous models like GPT-4. Cursor's standout features, such as Copilot++, Instant Apply, and Smart Rewrites, enable developers to seamlessly edit and navigate their codebases using natural language inputs. The integration of Fireworks' custom-trained model, specifically fine-tuned for the "Fast Apply" coding task, empowers Cursor to handle large-scale code edits with low latency, achieving speeds of around 1000 tokens per second. This cutting-edge approach to speculative decoding not only addresses the inefficiencies of traditional large language models but also positions Fireworks as a leader in providing high-speed, reliable AI solutions for developers and enterprises.

FireAttention V2 significantly enhances the performance of long context language models (LLMs), making them more practical for online inference, particularly for contexts ranging from 8K to 32K tokens. The Fireworks team has achieved major improvements, such as supporting FP16 and FP8 prefill kernels and introducing multi-host deployment modes beneficial for high-traffic applications. The post critiques existing benchmarks for long contexts, advocating for more comprehensive tests that require reasoning abilities beyond simple retrieval tasks. Benchmarking results show that the open-source Qwen 72B model is effective for long context tasks, while proprietary models also perform well. FireAttention V2 demonstrates superior throughput and latency compared to vLLM, particularly in FP8 mode, across both short-medium and long-generation scenarios. The multi-host mode further amplifies these gains, offering significant improvements in throughput and latency for enterprise customers.

Firefunction-v2 is an advanced open-source function calling model developed by Fireworks, designed to outperform existing models like GPT-4o in real-world scenarios by offering similar capabilities at a fraction of the cost and with significantly lower latency. The model integrates the robust multi-turn conversation capabilities of Llama 3 while excelling in function calling tasks, especially parallel function calling, which is crucial for intuitive user experiences and broader API usage. Unlike other open-source models, which often sacrifice general reasoning abilities for function specialization, Firefunction-v2 maintains a balance between function calling and general conversation tasks, making it adaptable to diverse applications. This is achieved through careful fine-tuning of Llama3-70b-instruct, preserving its instruction-following abilities while enhancing its function calling capabilities. Evaluations have shown that Firefunction-v2 consistently outperforms its predecessors and competitors in multiple benchmarks, demonstrating its efficacy in both function calling and non-function calling tasks. The model is available on the Fireworks platform, offering an easy transition for developers currently using OpenAI APIs, and is supported by a community-driven development approach that encourages feedback and continuous improvement.

Fireworks is revolutionizing the deployment of generative AI models by offering a highly configurable and cost-effective on-demand platform that leverages custom models and advanced hardware like H100 GPUs. This platform enables developers to import models from Hugging Face, scale deployments automatically, and optimize performance for various prompt sizes, all while reducing latency and costs compared to traditional solutions like vLLM. With features such as auto-scaling from zero and personalized serving stack configurations, Fireworks provides a seamless and efficient experience for businesses looking to scale their AI capabilities without long-term commitments. The platform's enhancements ensure that users can achieve the fastest, most reliable, and cost-effective AI model serving, catering to a wide range of use cases from start-ups to large enterprises.

Fireworks offers a range of GPU deployment options to support AI startups in scaling their operations, including serverless, on-demand, and enterprise solutions. On-demand GPUs provide a middle ground for companies needing reliable and fast processing without long-term commitments, offering significant cost and performance advantages over serverless and other platforms. Fireworks' on-demand service allows for automatic scaling, reducing idle costs, and offering flexibility in model choice and GPU configuration. The FireAttention stack enhances efficiency, providing substantial latency and throughput improvements. As companies grow, transitioning from serverless to on-demand becomes economically viable, with further potential to eventually adopt enterprise-level reserved GPUs for fully customized setups.