Nobody Is Blocked

This protocol replaces the second blocking acquire with a trylock, and it does remove deadlock: no thread ever holds one lock while waiting on another.

top:
    pthread_mutex_lock(L1);
    if (pthread_mutex_trylock(L2) != 0) {
        pthread_mutex_unlock(L1);
        goto top;
    }
    /* both held: do the work */

Thread 2 runs the same loop with L2 and L1 swapped. What can still go wrong?

A

Nothing: this is the standard deadlock-free ordering protocol and its only cost is the extra trylock call.

B

Deadlock comes back as soon as a third thread joins, since the protocol only breaks cycles of length two.

C

The two threads can grab their first lock, fail the trylock, release, and retry in lockstep indefinitely. Neither is blocked and neither makes progress.

D

pthread_mutex_unlock(L1) from a thread that reached it through a failed trylock is undefined behaviour, since the mutex was not acquired through pthread_mutex_lock in that iteration.