BibTeX download is not available in the pre-proceeding
While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obscures the need for rigorous evaluation of their reliability. In real-world deployment, however, achieving high reliability, where each added "nine" (e.g., "four-nines" (99.99\%) vs.\ "three-nines" (99.9\%)) is critical, as this gap yields an order-of-magnitude increase in failures, catastrophic in reliability-critical applications. Still, estimating such a rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes, making standard Monte Carlo evaluation infeasible under limited compute budgets. In this paper, we observe that, on parameterized GSM8K math-reasoning tasks, LLM failures exhibit strong systematic patterns: across these parameterized input spaces, a small subset of inputs disproportionately accounts for the majority of failures. Leveraging this observation, we propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method (CEM). We evaluate our framework on three LLMs, Qwen2.5-Math-7B-Instruct, gpt-oss-20b-low, and Gemini 2.5 Flash Lite, across parameterized GSM8K templates and achieve up to 156.22$\times$ reduction in required inferences compared to naive uniform sampling. Our estimates reveal that models with indistinguishable accuracy on standard benchmarks can differ substantially in estimated failure rates, underscoring that reliability is a distinct and measurable axis of model quality. Our simple framework takes a first step toward evaluating high-reliability regimes of LLMs on saturated GSM benchmarks, a distinct and underexplored dimension of evaluation beyond existing benchmarks.