Evaluating and Mitigating Misgendering in English-to-Hindi Machine Translation

Abstract

Machine translation between gender-neutral languages (those lacking grammatical gender on verbs and adjectives, such as English) and gender-marked Indian languages poses a fundamental challenge: systems must infer and realize grammatical gender in the target language, yet often default to masculine forms, rely on occupational stereotypes, or produce grammatically valid translations that obscure gender altogether. While gender bias in machine translation has been studied extensively for European languages, Indian language models remain under-evaluated, and existing benchmarks do not adequately capture the linguistic and cultural phenomena relevant to Indian settings. We introduce a 37,345-instance evaluation benchmark for English-Hindi gender bias spanning twelve linguistically motivated categories covering a broad range of gender-signaling phenomena, from stereotype inference and name-based disambiguation to cross-sentence gender tracking and Winograd-style coreference. Paired with a grammar-aware Hindi gender classifier combining rule-based morphological analysis with LLM fallback, the benchmark enables evaluation of six translation systems: three specialized MT models (Helsinki Opus-MT, NLLB-200, IndicTrans2) and three general-purpose LLMs (Sarvam AI, GPT-4o-mini, Claude Haiku 4.5). Across all six systems, we observe a consistent tendency to neutralize gender through Hindi's ergative construction, a grammatical pattern in which the subject takes the postposition ne and the verb agrees with the object, erasing subject gender. In our benchmark, 63% to 76% of translations use such syntax even when source-side gender is unambiguous, causing downstream gender classification accuracy to fall to just 17%–31% on sentences where gender should be recoverable. Accuracy ranges from 48.0% for Helsinki Opus-MT to 59.2% for Claude Haiku 4.5 at baseline. A cross-model prompting sweep across six strategies shows that prompting reduces overt masculine defaulting without eliminating ergative neutralization, with Haiku 4.5 reaching 74.6% at the strongest prompting level. Combining prompting with in-translation reranking further mitigates ergative neutralization, raising accuracy to 88.2% for Haiku 4.5 and 74.5% for GPT-4o-mini; candidate-selection reranking yields consistent gains for the specialized MT models. These findings position Hindi as a qualitatively distinct setting for gender bias evaluation and motivate targeting ergative syntax directly as the primary mitigation lever.