Fence In Local Memory

A compaction job takes a lease on a shard, then writes one output object. The lock service hands out a monotonically increasing fencing token per grant, and the object store is supposed to reject stale tokens. The store runs behind a load balancer across four stateless front-ends.

# job
token = lock.acquire("shard-7")       # returns 41, then 42, then 43, ...
blob  = compact(read_shard("shard-7"))
store.put("shard-7/out.parquet", blob, token=token)

# object store front-end
highest_seen = {}                      # process-local dict

def put(key, body, token):
    if token < highest_seen.get(key, 0):
        raise StaleToken()
    highest_seen[key] = token
    write_to_disk(key, body)

Job A holds token 41, stalls for 30 seconds in a GC pause, and its lease expires. Job B takes token 42 and writes through front-end 2. Job A then wakes and writes through front-end 3. Front-end 3 accepts it. Where is the bug?

The lease is too short for the observed GC pauses. Raising it above the worst pause the fleet has recorded prevents the overlap that makes the stale write possible.

highest_seen is per-process, so each front-end keeps its own maximum and front-end 3 has never seen 42. The comparison has to happen at one serialization point for the object, as a conditional write against a token stored with the object itself.

The comparison is < where it should be <=, so a token equal to the highest seen is let through and job A's retry slips in.

The job never re-checks the lease after compact() returns. Calling lock.still_held("shard-7") immediately before the put closes the window.