Which tokens does a hybrid model predict better?

The study examines the prediction capabilities of hybrid language models compared to standard transformers, focusing on token-level differences. Conducted with the Olmo Hybrid and Olmo 3 models, the research reveals that hybrids excel in predicting meaning-bearing tokens such as nouns, verbs, and adjectives, as well as tokens requiring contextual understanding, like pronouns. However, their advantage diminishes on repeated tokens, where transformers excel due to their attention mechanism's ability to recall specific earlier tokens. The hybrid model's strength lies in its recurrent layers' ability to track state changes, though it struggles with precise recall. The findings suggest that evaluating models based on specific token types rather than overall loss provides a clearer picture of architectural strengths and weaknesses, particularly highlighting the hybrid's proficiency with open-class tokens. The research encourages deeper exploration of token-specific losses to enhance model development, with the aim of refining hybrid architectures by understanding each component's unique capabilities.