Whoever Touches It First

A dual-socket Linux box, 24 cores per socket. This loop runs at half the memory bandwidth the machine can do, and perf attributes most of the traffic to the inter-socket link when threads on socket 1 run.

double *a = malloc(N * sizeof(double));   /* N = 2e9 */
memset(a, 0, N * sizeof(double));         /* main thread, before the parallel region */

#pragma omp parallel for
for (long i = 0; i < N; i++)
    a[i] = a[i] * 2.0 + 1.0;

What put the data in the wrong place?

The array is far larger than either socket's L3, so every access misses cache and the inter-socket link is simply the visible cost of that.

malloc commits physical memory on the node of the calling thread, so the placement was decided by the allocation call and nothing later can change it.

The OpenMP runtime migrates worker threads between sockets between iterations, so half the accesses are remote no matter where the pages live.

The default policy allocates each page on the node of the CPU that first writes to it, and the single-threaded memset first-touched every page from a core on socket 0.