What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization

Abstract

In domains like medicine and finance, large-scale labeled data is costly and often unavailable, leading to models trained on small datasets that struggle to generalize to real-world populations. Large language models contain extensive knowledge from years of research across these domains. We propose LoID (Logit-Informed Distributions), a deterministic method for extracting informative prior distributions for Bayesian logistic and linear regression by directly accessing their token-level predictions. Rather than relying on generated text, we probe the model's confidence in opposing semantic directions (positive vs. negative impact) through carefully constructed sentences. By measuring how consistently the LLM favors one direction across diverse phrasings, we extract the strength and reliability of the model's belief about each feature's influence. We evaluate LoID on 15 real-world tabular datasets under synthetic out-of-distribution (OOD) settings characterized by covariate shift. We compare our approach against (1) standard weakly informative priors, (2) AutoElicit, a method that prompts LLMs to generate expert-style priors, (3) LLMProcesses, a method that prompts LLMs to generate predictions and (4) an oracle-style upper bound derived from fitting logistic/linear regression on an in-distribution (IID) train set. We assess performance using Area Under the Curve (AUC) and Mean Squared Error (MSE). Across datasets, LoID significantly improves performance over logistic and linear regression models trained on OOD data, recovering \textbf{42\%} of the performance gap relative to the oracle model on average.