Semantic Cache

The 2023 GPTCache paper described an open-source architecture with embeddings, similarity evaluation, vector storage, and response reuse. AWS later documented the same request-response pattern for managed vector search. Recent Apple research separates curated static answers from dynamically populated cache entries and studies verification near the similarity threshold, showing that production work is moving from simple nearest-neighbor lookup toward explicit quality policies.

Sources: s1, s2, s3

Repeated questions can trigger expensive model inference even when an acceptable answer already exists. A semantic cache can reduce calls and latency across paraphrases, especially for stable support or knowledge tasks. Unlike exact caching, it can also return the wrong answer when two requests look similar but differ by negation or another answer-changing detail. That makes hit precision and invalidation first-class evaluation targets.

Sources: s1, s2, s3

An internal IT assistant stores a VPN-installation question, its embedding, and the approved answer. A paraphrased request can reuse that answer when its similarity score clears the chosen threshold; otherwise the application calls the model. The team tests the cache on paraphrases and answer-changing near-matches, measures incorrect reuse separately from miss rate, and refreshes entries when the underlying instructions change.

Sources: s1, s2, s3

Embedding similarity is not equivalence. A cache can suppress a needed fresh model call or preserve an answer after its underlying information has changed. Very conservative thresholds may erase the economic benefit, while aggressive thresholds increase incorrect reuse. Operators should choose thresholds against task-specific quality targets, verify borderline matches, measure hit quality alongside latency and cost, refresh stale entries, and keep a miss or review path for uncertain or consequential requests.

Sources: s1, s2, s3