Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

Abstract

We introduce WIKI-BLOOM, a multilingual diagnostic benchmark for evaluating how large language models reason over supplied relational knowledge, with the goal of separating evidence-conditioned reasoning from memorized entity associations. WIKI-BLOOM contains 450 multiple-choice questions grounded in 30 curated Wikidata relations, instantiated under factual, counterfactual, and neutralized conditions and spanning five levels adapted from Bloom's revised taxonomy: Remember, Understand, Apply, Analyze, and Evaluate. We evaluate 43 models, 41 open-weight models across eight series and two closed-weight Gemini models, in six languages (Arabic, Chinese, English, Hindi, Persian, and Spanish) and two multilingual settings, yielding 226,800 predictions. Statistical inference is clustered at the level of the 30 source relations. Several concrete patterns emerge. Open-weight families consistently exhibit an Apply-to-Analyze accuracy drop, but its magnitude varies substantially with family and scale and disappears entirely for both Gemini models. Evaluate, however, remains a persistent bottleneck even when the earlier cliff has closed. Gemini 3.1 Pro, for example, averages 99.5\% across Remember, Understand, Apply, and Analyze, but only 60.7\% on Evaluate. Item-difficulty rankings are significantly stable across languages in every evaluated family and size bucket, and relation-difficulty hierarchies show similarly strong cross-language stability. The interaction between parametric knowledge and supplied context also changes with cognitive demand. The Neutralized-over-Factual advantage is often larger at Analyze and Evaluate than at Remember and Understand, with gains reaching more than 20 percentage points in some settings, suggesting that entity-specific associations become increasingly disruptive as reasoning demands increase. Counterfactual contexts are generally the weakest condition, and the factual--counterfactual gap widens with cognitive complexity in several model families, including both Gemini models, showing that context conflict can remain damaging even near ceiling performance. Together, these results show that increasing model capability alone does not eliminate failures in cognitively demanding use of supplied knowledge, and that the structure of these failures is remarkably stable across languages.