MURMUR: Cross-Lingual and Multimodal Retrieval-Augmented Reasoning for Open Question Answering in Tamil and Yoruba

Abstract

As large language models with retrieval-augmented generation gain traction in multimodal knowledge base question answering, concerns about their transfer to low-resource languages remain unaddressed. We introduce MURMUR, a benchmark evaluating multimodal cross-lingual retrieval and reasoning in Tamil and Yoruba. Using the hardest examples from WebQA and MultimodalQA, we build a high-quality low-resource language benchmark through LLM-assisted translation, human validation, and culturally aligned rewriting that reflects native-speaker phrasing (i.e., what a native speaker would naturally ask) while preserving answerability. MURMUR exposes major performance gaps and failure modes in current systems. We provide XM-RAG as a supporting cross-lingual multimodal RAG baseline, which reaches 38.1 answer accuracy and exceeds the next-best baseline by more than 6.3 points. XM-RAG still performs far below top English results on WebQA and MultimodalQA, showing that existing methods struggle with complex tasks in low-resource language settings. By releasing MURMUR with a reproducible baseline, we offer a resource for measuring these gaps and guiding progress toward equitable multimodal knowledge base question answering.