Large Language Model is not a (Multilingual) Compositional Relation Reasoner

Jinman Zhao, Xueyan Zhang

Conference On Language Models 1 pre-proceedings (COLM 2024) Conference

BibTeX download is not available in the pre-proceeding

We present a comprehensive evaluation of large language models' capability to reason compositional relations through a benchmark encompassing 1,800 test cases in both English and Chinese, covering six distinct categories of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. We expand our assessment to the multilingual realm by including translations of the benchmark suite into Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling compositional relation reasoning across diverse linguistic contexts.