June 2023 Summaries

In the AI industry, infrastructure challenges often lead to reliance on simple metrics like QPS, Latency, and Throughput, resulting in tools that excel in benchmarks but struggle in practical deployments. A critical yet overlooked feature is support for dynamic shapes, which significantly impacts real-world performance and usability. The Modular AI Engine offers full support for dynamic shapes, allowing it to outperform traditional compilers like XLA on Intel CPUs, especially when running BERT on the GLUE dataset, achieving 5x faster compile times and 2x faster runtime. This engine supports various models from popular frameworks and combines the usability of dynamic execution with the performance of a compiler approach, unlike static compilers that require cumbersome workarounds like padding for sequence lengths. Hybrid approaches exist, but the Modular AI Engine's dynamic compiler provides superior performance without the need for such mitigations. It offers a seamless experience, reducing costs and latency while simplifying the deployment process for AI developers.

Mojo is positioned as a promising programming language for the future of AI-assisted development, particularly in the context of large language models (LLMs) like Copilot and Ghostwriter. While LLMs are enhancing developer workflows by generating and reviewing code, programming languages remain essential for clear human-to-human communication and precise instructions to computers. Mojo aims to combine the human readability of Python with advanced system programming features, allowing it to scale across various hardware and applications. It is designed to be a fully compatible superset of Python, benefiting from its extensive community and ecosystem, while also integrating seamlessly with C and C++ for expanded functionality. The language's development draws on decades of experience in building compilers and seeks to facilitate a new era of collaborative software development, making it an ideal candidate for LLM-generated code. Mojo's 0.1 release marks the beginning of its journey, inviting developers to contribute to its growth and evolution.

The Modular AI Engine has been introduced as a fast and versatile AI inference engine providing significant usability, portability, and performance improvements for AI frameworks like PyTorch and TensorFlow. Integrated into existing serving solutions such as NVIDIA's Triton Inference Server and TensorFlow Serving, it offers seamless deployment with features like dynamic batching and concurrency. Performance testing on various hardware, including AWS Graviton2, AMD EPYC, and Intel Skylake, demonstrated that the Modular AI Engine achieves superior throughput and lower latency compared to TensorFlow and PyTorch. By maintaining consistent default settings, the engine improves upon common models like BERT-base in production environments, highlighting its ability to scale efficiently across multiple architectures. As part of its ongoing development, Modular is enhancing its platform for performance-sensitive AI models and invites users to explore more details on its Performance Dashboard and stay updated through its newsletter.