BibTeX download is not available in the pre-proceeding
Calibrated confidence, where a model's stated probability matches its actual expected accuracy, is often essential in contexts where confidence scores are relied upon by humans to make decisions or serve as inputs to downstream agents. Existing approaches for training models to verbalize calibrated confidence typically require ground-truth correctness labels, which are costly to collect and unavailable for many prompt distributions. Multi-pass consistency methods achieve good calibration but significantly increase inference costs. We introduce CONCORD, a label-free method for training language models to improve the calibration of verbalized confidence scores without adding a dedicated calibration stage. Adapting models with Group Relative Policy Optimization already yields multiple rollouts per prompt; we repurpose these rollouts as a calibration signal by deriving confidence targets from semantic agreement among sampled answers, using either Natural Language Inference-based answer clustering or semantic entropy. The resulting reward encourages the model to align its stated confidence with empirical semantic agreement, without requiring ground-truth labels. During inference, the model generates a reasoning trace, an answer, and a better calibrated confidence score in a single pass. Across six benchmarks and four models, CONCORD substantially reduces expected calibration error over unadapted verbalized confidence and over label-dependent adaptation baselines, while maintaining competitive AUROC.