Question 1542

Tools Above The Breakpoint

POLICY_TEXT is 9,800 tokens and has not changed in a month. TOOL_USAGE_NOTES is 1,200 tokens and is also static. Yet cache_read_input_tokens comes back 0 on nearly every request.

def build_request(user, history, question):
    # surface the tools this user reaches for most, so the model sees them first
    tools = sorted(ALL_TOOLS, key=lambda t: -usage_count(user, t.name))
    return {
        "model": MODEL,
        "tools": tools,
        "system": [
            {"type": "text", "text": POLICY_TEXT},
            {"type": "text", "text": TOOL_USAGE_NOTES,
             "cache_control": {"type": "ephemeral"}},
        ],
        "messages": history + [{"role": "user", "content": question}],
    }

What is defeating the cache?

A

The cacheable region is 11,000 tokens, which is under this model's minimum cacheable prompt length.

B

tools sits above system in the cached prefix, and re-sorting it by a per-user counter changes the prefix on nearly every request.

C

The breakpoint is on TOOL_USAGE_NOTES, so only those 1,200 tokens are eligible and POLICY_TEXT is never cached.

D

messages grows every turn, and a change in messages invalidates the system blocks above it.