Semantic cache

The semantic cache answers a repeated question without contacting a provider at all: embed the prompt, cosine-match against stored entries (≥ 0.95 by default), and return the stored completion.

What a hit means

• $0 billed. The response’s cost says price_source: semantic-cache-hit.
• Savings are recorded, not just spend avoided: the ledger row carries tokens_saved and usd_saved, so the cache’s value shows up in the same reports as the spend it prevented. GET /v1/cache/stats aggregates it.
• A span is still emitted (with costhelm.cache.hit = true), so traces show the hit rather than a gap.

What is comparable

Two prompts are only comparable if the fields that change the answer match exactly: model, provider, system prompt, tools, response format, temperature, max_tokens, and routing role — all hashed into the cache namespace. Entries are scoped to principal dimensions you choose, so one tenant’s cache never answers another’s.

What is refused into the cache

• Truncated completions (stop_reason: max_tokens) — caching a cut-off answer would replay the mistake forever.
• Empty completions.
• Streaming responses — the full text isn’t assembled through the same path.
• High-temperature requests, unless the caller opts in per call ("semantic_cache": true).

Enabling it

Ships disabled. It needs an embedder — a local Ollama nomic-embed-text works and costs nothing:

~/.costhelm/cache.yaml

semantic:
  enabled: true

TTL, max entries, the similarity threshold, and principal scoping are all in the same file — see the configuration reference. If no embedder is available the cache fails soft: the gateway serves normally with the cache off, and says so in config_errors.