3 / 4 Attention & Transformers (intro)

Eight Eyes Are Better Than One

A transformer layer uses multi-head attention with 8 heads instead of a single attention operation over the full model dimension. What is the main benefit of splitting into multiple heads?

A

It removes the need for the softmax normalization step

B

Each head can learn to attend to different types of relationships or subspaces in parallel, giving richer representations

C

It reduces the total number of parameters compared to a single attention head

D

It guarantees the model converges faster on any dataset