What Makes Retrieval Work for Pedagogical Dialogue Annotation? Indexing Granularity over Retriever Adaptation

Abstract

Annotating tutoring dialogue for pedagogical moves requires trained coders and extensive calibration, which is a major bottleneck that limits the scale of research on tutoring quality. Retrieval-augmented prompting can ease this bottleneck while leaving the generative model frozen, but it is unclear whether the gain comes from the retriever, the segmentation of the dialogue, or the granularity of the exemplar index. We study retrieval-augmented prompting on two English mathematics tutoring datasets (TalkMoves and Eedi) that share a six-category tutor move taxonomy, using three LLM backbones (GPT-5.2, Claude Sonnet 4.6, and Qwen3-32B). On a matched subset where every condition returns a valid label, we find that retrieved demonstrations raise Cohen's $\kappa$ from $0.41$ to $0.55$ on TalkMoves and to $0.64$ on Eedi for the strongest backbone. Index granularity accounts for most of the improvement and retriever training and segmentation contribute little. Fine-tuning the embedder shifts $\kappa$ by $−0.04$ to $+0.05$, lexical BM25 matches dense retrieval to within $0.02$ everywhere, and semantic chunking does not beat a three-utterance fixed window. Moving the index unit from the chunk down to the individual utterance, with the parent chunk returned as context, contributes $+0.07$ to $+0.14$. These findings suggest that the unit of the index matters more than the training of the retriever for retrieval-augmented annotation of tutoring dialogue.