Charged Twice On Retry (2)

An agent pipeline runs on a durable workflow engine. Support reports that some users' prepaid balances drop twice for a single generation. It happens on maybe 2 percent of requests, and always on slow ones.

@activity.defn
async def generate_and_charge(user_id: str, prompt: str) -> str:
    text = await llm.complete(prompt)        # p99 latency 41s
    tokens = count_tokens(prompt, text)
    await billing.debit(user_id, tokens)     # INSERT into ledger, UPDATE balance
    return text

The activity is configured with a 45 second start-to-close timeout and the engine's default retry policy.

What is the correct fix?

Move the billing.debit call out of the activity and into the workflow function, where replay guarantees it runs exactly once.

Give the debit an idempotency key stable across attempts (workflow id plus activity id, not attempt number) and make the ledger insert conditional on that key.

Raise the start-to-close timeout to 120 seconds so the activity stops timing out on slow generations.

Set the retry policy to maximum_attempts=1 so a timed-out activity is never re-executed.