Late to the Party

A job holds a distributed lock with a 10 second lease before writing a file to object storage. The process stalls for 25 seconds in a stop the world pause, the lease expires, a second job takes the lock and writes, and then the first job wakes up and issues its own write. Which fix actually prevents the stale overwrite?

Keep every node's clock within a few milliseconds using NTP so both jobs compute the same expiry instant for the lease

Have the lock service issue an increasing fencing token per grant and have storage reject any token below the highest it has seen

Raise the lease duration to 60 seconds so that it comfortably exceeds the longest stop the world pause the team has ever observed

Have the client re-read the remaining lease time immediately before issuing the write and abort if less than a second is left