Prompt caching

How cache hits work#

Caching matches on the prefix: starting from the beginning of the request, everything that is byte-for-byte identical can hit the cache. A change anywhere invalidates everything after it. So put stable content first (system prompt, tool definitions, reference documents) and changing content last (the user's new question, timestamps and so on).

GPT models: automatic caching#

GPT models cache long request prefixes automatically, with no code changes. Cached tokens are reported in usage.prompt_tokens_details.cached_tokens:

"usage": {
  "prompt_tokens": 12400,
  "completion_tokens": 310,
  "prompt_tokens_details": { "cached_tokens": 11264 }
}

Claude models: mark breakpoints with cache_control#

With Claude, you mark a cache breakpoint with cache_control on a content block. Everything before it (tools → system → messages) is cached:

from anthropic import Anthropic

client = Anthropic(api_key="sk-...", base_url="https://api.flyrouter.ai")

long_document = open("manual.md").read()  # a long reference document that never changes

msg = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=2048,
    system=[
        {"type": "text", "text": "You are a support agent. Answer only from the manual below."},
        {"type": "text", "text": long_document, "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": "How do I reset the device?"}],
)
print(msg.usage)

• The cache lasts 5 minutes by default, and every hit resets the timer. For longer gaps between requests, use {"type": "ephemeral", "ttl": "1h"} for a 1-hour cache.
• A request can have at most 4 cache breakpoints.
• Content that's too short isn't cached (about 512 tokens minimum on newer Claude models, more on older ones), and no error is raised.

Two usage fields show whether it's working:

If cache_read_input_tokens stays at 0 across several requests, something that changes on every request has usually slipped into the prefix.

Billing#

• Cache hits are billed at the cache-read price, far below the normal input price.
• Cache writes (Claude) are billed at the cache-write price the first time, slightly above the normal input price; 1-hour cache writes cost more.
• Cache read and write prices for every model are on the model pricing page.

If a piece of content is reused two or more times, caching almost always pays off.

Common reasons the cache misses#

• The system prompt contains the current time, a request ID or something else that changes every time.
• The tool list changes in order or content between requests.
• JSON is serialized with keys in a non-deterministic order.
• You switched models. Caches are per model, so a new model has to write its own.