BibTeX download is not available in the pre-proceeding
Recent approaches to discourse coherence assessment increasingly rely on language models, but it remains unclear whether they evaluate underlying discourse structure or merely surface-level lexical patterns. We use metaphor as a controlled lexical-semantic and stylistic probe. Using a paraphrase framework with automatic quality control and small-scale human validation, we generate metaphorical rewrites that hold propositional content fixed while varying local lexical realization. Across nine models spanning three architecture families, we find that models paradoxically reward figurative language, assigning higher coherence scores to texts with novel metaphors in nearly all model-domain conditions. Their internal representations appear to encode whether a rewrite changed the meaning, suggesting that the relevant information is present but is not what drives the judgment. Coherence judgments are thus influenced by local stylistic variation, indicating that current models often evaluate linguistic realization rather than relying on discourse-level reasoning alone.