Prompt caching and saving cost

In short: With prompt caching the provider remembers the already computed processing of an unchanged start of a prompt. Later requests with the same start are then faster and much cheaper.

How it works

While reading a prompt the model computes intermediate values for every token, the KV cache (see Self-attention in detail). If the start of two requests is identical, the provider can reuse these values instead of recomputing them.

• Only the same beginning (prefix) is cached. If even the first character changes, the cache does not apply.
• So put the constant part (system instruction, long documents, tool descriptions) first and the changing part (the current question) last.
• Cached parts are often kept for only a few minutes and must have a minimum length. Each provider sets details and prices itself.

More ways to save

1. A smaller model for simple tasks, a larger one only when needed.
2. Ask for shorter outputs, because generated tokens usually cost more than read ones.
3. Batch processing for non-urgent jobs.
4. Less context: send only the document parts that are really needed, see Vector databases and similarity search.

See also: Inference servers: batching, PagedAttention and vLLM