Cache semantically similar prompts instead of paying twice

The same question comes in worded five different ways, and my cache only ever catches the exact repeats.

IntermediateRead 7 minDoing: a day7 steps

▶ Start

What you'll have at the end

A cache hit rate measured before and after, with the bill dropping to match

Before you start

You need

You already have a cache that returns a stored answer when a new prompt is byte-for-byte identical to one you answered before, and a way to turn text into a numeric vector on demand.

Not covered

Building your first exact-match cache, and provider-side prompt caching (the discount for reusing a repeated prefix inside one request), are not covered here.

Leans on

LLMOps Cheat SheetNames semantic caching as a cost lever, the general idea this recipe turns into a working layer.
Multi-Model Routing and LLM Gateways Cheat SheetCovers the hash-plus-semantic layering and threshold tuning this recipe wires together end to end.

7 steps

Where to go next

Route your easy requests to a cheaper model

When questions genuinely differ and nothing repeats much, routing cheap requests to a smaller model cuts cost without the wrong-answer risk a cache carries.

Checked 15 Aug 2026

Part of the LLMOps cookbook