BibTeX download is not available in the pre-proceeding
Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce an evaluation protocol built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on ten models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate, and counterfactual controls reveal that the models respond to the presence of prompt text rather than to its content. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. The residual benefit of context concentrates on the hardest utterances, and a difficulty-aware gate that applies clinical context only where it helps lowers WER by about 10% relative over the audio-only baseline on both architectures. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.