Three Layers Of Averaging

A three-node path graph $A - B - C$ carries scalar node features. Each layer replaces every node's feature with the mean over itself and its neighbours:

$h_v \leftarrow \text{mean}\big({h_v} \cup {h_u : u \sim v}\big)$

so $A$ averages over ${A, B}$ , $B$ over ${A, B, C}$ , and $C$ over ${B, C}$ . Start from $h^{(0)} = (1, 0, -1)$ and apply three layers. What are the features, and what does the pattern mean for a 10-layer version of this network?

$(0,\ 0,\ 0)$ after a single layer, since averaging over a connected graph reaches its fixed point immediately.

$(0.125,\ 0,\ -0.125)$ : the gap between $A$ and $C$ halves at every layer and tends to 0, so a deep stack drives all node representations toward the same vector and a classifier can no longer separate the nodes.

$(0.125,\ 0,\ -0.125)$ : the shrinking activations are vanishing gradients, so the fix is a larger learning rate or a non-saturating activation.

$(0.5,\ 0,\ -0.5)$ : the first layer smooths, and further layers recompute the same average from the original features, so the values stop changing.