MoANT: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning

Abstract

Large language models (LLMs) encounter significant challenges when adapting to diverse multitask fine-tuning settings. Mixture-of-experts (MoE) offers a promising solution with a dynamic architecture that enables effective task decoupling. However, scaling up the number of MoE experts incurs substantial parameter and computational overhead and yields limited performance gains due to naive routing mechanisms. In this paper, we design a novel framework, Mixture-of-rAnk-oNe-experTs (MoANT), which leverages the combination of ultra-low-rank experts to construct LoRA weights on pretrained LLMs. The rank-1 experts enable a mix-and-match mechanism that quadratically expands the vector subspace of experts with linear parameter overhead, achieving approximate error reduction with optimal efficiency. In addition, MoANT offers implicit guidance to the router by leveraging the inherent semantic clustering of training embeddings as prior knowledge, enabling optimized feature allocation across experts for smoother convergence. Extensive theoretical and empirical results demonstrate that MoANT achieves superior efficiency and generalizability across a diverse range of tasks. Compared with traditional LoRA, MoANT achieves up to 1.78% mean accuracy improvement with around 30%-40% fewer trainable parameters across 14 public datasets.