Aether-7B-5Attn: A 100% Open-Source Sovereign Foundation Model — and a Controlled Experiment in Heterogeneous Attention

Aether-7B-5Attn, developed by the Korean AI startup VIDRAFT, is an open-source language model that challenges the conventional uniform attention mechanism in transformer layers by employing a heterogeneous attention approach, distributed across its 49 layers in a Latin square arrangement. Released under Apache-2.0, it offers complete transparency with access to the training data, code, logs, and checkpoints, setting a standard for sovereignty in AI development by enabling full reproducibility and verifiability. The model, which employs a Mixture-of-Experts architecture with 6.59 billion parameters, is designed to test whether using different attention mechanisms in various layers enhances performance. Aether's approach argues that true AI sovereignty comes from the ability to rebuild and modify the model independently, distinct from merely downloading existing model weights. This initiative not only contributes to the scientific exploration of attention mechanisms but also democratizes the model-building process, allowing other developers to iterate and innovate upon its foundation.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.