Cache Hit Rate

In depth

For AI-powered SaaS, cache hit rate is the single most important unit-economics lever. The top 500-2,000 queries in almost every vertical follow a long-tail distribution — they cover 60-80% of total query volume. Caching those once and serving them from Redis or Postgres costs almost nothing, while running them through an LLM costs real money per call.

The target cache hit rate depends on the product. Low-variance verticals (exam prep, common customer-support questions, FAQ lookup) can reach 70%+ steady-state. High-variance verticals (bespoke research, creative generation) cap lower, around 30-40%. A product priced at Rs 99 per month with an LLM cost ceiling of Rs 25 per MAU almost always requires at least 40% hit rate to be profitable.

Beyond cost, cache hits are 50-200x faster than fresh LLM calls. Users experience a materially faster product, which improves retention directly.

Formula & example

Rules of thumb

• Target 40%+ hit rate within three months of shipping; 70%+ is achievable for low-variance domains.
• Cache by canonical-hash, not by raw query string — normalization catches paraphrases.
• Monitor hit rate weekly; a sudden drop usually signals a shift in user behavior worth investigating.
• Pre-populate the cache with SME-verified answers for the top 500 queries before launch.

Common mistakes

• Cache with no invalidation — stale answers eventually embarrass the product.
• Serving cache hits with no quality checks — the cache is only as good as what populated it.
• Optimising cache hit rate after shipping LLM costs blow the budget — retrofitting is expensive.

Put it into practice

FAQ

Related terms