Tiny Aya: Bridging Scale and Multilingual Depth

Alejandro R Salamanca, Diana Abagyan, Daniel D'souza, Ammar Khairi, David Mora, Saurabh Dash, Viraat Aryabumi, Sara Rajaee, Mehrnaz Mofakhami, Ananya Sahu, Thomas Euyang, Brittawnya Prince, Madeline Smith, Hangyu Lin, Acyr Locatelli, Sara Hooker, Tom Kocmi, Aidan Gomez, Ivan Zhang, Phil Blunsom, Nicholas Frosst, Joelle Pineau, Beyza Ermis, Ahmet Üstün, Julia Kreutzer, Marzieh Fadaee

Conference On Language Models 3 pre-proceedings (COLM 2026) Conference

BibTeX download is not available in the pre-proceeding

We introduce Tiny Aya, a family of open-weight multilingual language models that achieve strong, balanced performance across 70 languages with 3.35 billion parameters through deliberate data curation and training design. The release includes a pretrained base model, a globally balanced instruction-tuned variant, and three region-specialized models targeting Africa, South Asia, Europe, Asia-Pacific, and West Asia. We contribute: (1) a script-aware tokenizer weighting scheme that achieves competitive or superior compression across most writing systems, particularly for underrepresented scripts; (2) a posttraining pipeline combining FusioN, a multi-teacher synthetic data generation framework, with region-aware supervised finetuning and SimMerge, a predictive model merging method that preserves global consistency while retaining regional gains; (3) a comprehensive multilingual evaluation suite spanning translation, understanding, reasoning, generation, safety, and cultural awareness across up to 66 languages. Despite its compact size, Tiny Aya outperforms Gemma3-4B on translation in 46 of 55 languages, is competitive with or outperforms similar-sized models on generative benchmarks with substantially lower cross-language variance, and achieves the highest mean safe response rate (91.1%) on multilingual safety benchmarks. Quantized variants run on smartphones and quantization degradation attenuates for lower-resource languages . We release all models to support research into efficient, inclusive multilingual AI.