Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources—n-gram matches copied from the input context, and statistical predictions from prior forward passes—differ sharply in acceptance rate (~6x median gap, range 2–18x across five models and five benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable tokens should form a deep chain while unreliable tokens spread as wide branches, raising the depth ceiling of balanced trees. We realize this structure in Goose, a training-free framework that builds an adaptive spine tree—a deep chain of high-acceptance context-matched tokens with wide branches of low-acceptance alternatives at each node. The resulting tree provably accepts at least as many tokens per step as either source alone. On five LLMs (7B–33B) and five benchmarks, Goose achieves 1.9–4.3x lossless speedup, outperforming balanced-tree baselines by 12–33% under the same budget.