Prompt Caching in Claude & Gemini: How to Cut LLM Token Latency and Costs by up to 90%

As enterprise AI applications evolve from simple conversational chatbots into complex autonomous agents with massive system prompts, tool definitions, few-shot examples, and extensive RAG documentation, API costs and time-to-first-token (TTFT) latency have become the primary bottlenecks of production deployments. Sending 50,000 to 100,000 tokens of context on every single API turn is both financially unsustainable and frustratingly slow. Prompt Caching—implemented natively in Anthropic’s Claude API and Google’s Gemini models—solves this architectural challenge by caching the pre-computed Key-Value (KV) attention states of repetitive prefix tokens on the provider’s GPU clusters, reducing costs by up to 90 % and slashing latency by over 80 %.

Under the Hood: How Transformer KV-Cache Reuse Works

To appreciate why prompt caching delivers such dramatic performance gains, one must understand how autoregressive transformer models process input tokens:

During the prefill phase of an LLM request, the model calculates the Key ($K$) and Value ($V$) projection matrices for every token across all transformer layers. For a 100,000-token prompt, calculating the self-attention matrix requires massive compute ($O(N^2)$ attention operations). Without caching, the provider must recompute the entire matrix from scratch for every single message in a multi-turn conversation.

With Prompt Caching, the inference engine hashes the exact token sequence of the prompt prefix. If an identical prefix was processed recently, the model bypasses the prefill phase entirely and loads the pre-computed KV tensors directly from ultra-fast GPU HBM memory. The model only computes attention for the newly appended delta tokens.

Claude vs. Gemini: Implementation Differences and Pricing Dynamics

Architectural Best Practices: Structuring Prompts for Maximum Cache Hits

Prompt caching operates strictly from left to right. Even a single modified character, whitespace difference, or timestamp placed at the beginning of a prompt invalidates the entire cache for all subsequent tokens. To maximize your cache hit ratio in production, enforce the following structural order:

1. Static System Instructions (Cached): Role definition, architectural guidelines, tool JSON schemas, and formatting rules. This block should remain 100 % invariant across sessions.
2. Static Reference Data & Documentation (Cached): Full API specifications, database schemas, and few-shot examples. Mark the end of this block with your primary cache breakpoint.
3. Conversation History (Partially Cached): In multi-turn chat sessions, attach a secondary cache breakpoint to the second-to-last user turn.
4. Dynamic User Input (Never Cached): The latest user query, dynamic timestamps, and session-specific runtime variables must strictly reside at the very end of the prompt.

Production Example: Python SDK Implementation with Anthropic

import anthropic

client = anthropic.Anthropic()

# Define a massive 20k token system prompt with a cache breakpoint
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are an expert distributed systems architect...",
        },
        {
            "type": "text",
            "text": "[... 50,000 TOKENS OF SYSTEM SPECS AND DOCUMENTATION ...]",
            "cache_control": {"type": "ephemeral"}  # Breakpoint 1
        }
    ],
    messages=[
        {"role": "user", "content": "Analyze the consensus protocol in Section 4."}
    ]
)

# Inspect token savings in production telemetry
print(f"Tokens written to cache: {response.usage.cache_creation_input_tokens}")
print(f"Tokens read from cache:    {response.usage.cache_read_input_tokens}")
print(f"Regular input tokens:      {response.usage.input_tokens}")

Frequently Asked Questions (FAQ)

Does prompt caching affect the output quality or determinism of the LLM?

No. Because the KV-cache contains the exact floating-point representations of the attention layers, the mathematical output of the transformer is completely identical whether a prompt is computed from scratch or served from cache.

What happens if a cached turn expires after 5 minutes of user inactivity?

If a user takes longer than 5 minutes to respond in Claude, the cache entry expires. The subsequent API call will automatically re-write the cache (incurring the 25 % cache creation premium), and all following turns will immediately benefit from cache hits again.