TierCache: A Multi-Granularity Semantic Caching Framework for Structured Query Generation

Abstract

Semantic caching is a promising solution to reduce LLM serving costs by reusing cached responses. However, existing methods are predominantly designed for single-level, verbatim response reuse. This limits their effectiveness in structured query generation, where queries often share output structures. To address this, we propose **TierCache**, a multi-granularity semantic caching framework that introduces two-stage retrieval and two-tier reuse: Tier 1 provides the fastest path via semantic retrieval and verbatim reuse, while Tier 2 enables the reuse of structurally similar queries through structure-aware retrieval and lightweight slot filling. We evaluate our approach on two structured query generation tasks: Text-to-SQL and Text-to-Cypher. To this end, we construct **SQStream**, a semantic caching benchmark comprising ten query-stream workloads–six for SQL and four for Cypher–that preserve realistic cache-reuse opportunities. Across all workloads, TierCache consistently establishes the most favorable quality-cost Pareto frontier. Under a strict quality-preserving constraint ($>95\%$ retention), it achieves 1.28$\times$–4.44$\times$ the cache hit rates of state-of-the-art semantic caching baselines, delivering 1.48$\times$–2.99$\times$ end-to-end speedups. We release our code and benchmark at https://github.com/myeon9h/TierCache.