Sixteen Bytes Apart

Two threads pinned to different cores each hammer one field of this struct ten million times. The cache line is 64 bytes and the struct is 64-byte aligned.

struct counters { long a; long b; };   /* 16 bytes */
static struct counters c;

void *worker_a(void *_) { for (long i = 0; i < 10000000; i++) c.a++; return 0; }
void *worker_b(void *_) { for (long i = 0; i < 10000000; i++) c.b++; return 0; }

Both final values are exactly correct, but the program runs several times slower than the same loops on two separate structs. Why?

The struct straddles a page boundary, so each write costs a TLB lookup on top of the cache access.

a and b sit in one 64-byte line, so each core's write puts the line in Modified state and invalidates the other core's copy; the line ping-pongs across the interconnect on every increment.

long increments are not atomic, so the coherence protocol has to serialize the two cores' read-modify-write sequences against each other.

The two loops keep both cores' prefetchers streaming the same address range, saturating memory bandwidth.