1 / 10 Regularization & Loss Techniques

Team Norm

A Transformer block normalizes activations per-token across the feature dimension, independent of other examples in the batch, so it works fine even with batch size 1 or variable-length sequences. Which normalization technique is this, and why is it preferred over BatchNorm here?

A

BatchNorm, because it uses running averages that make it insensitive to batch size

B

LayerNorm, because it computes statistics per sample across features rather than across the batch, avoiding batch-size and sequence-length dependence

C

BatchNorm, because it computes statistics per sample across features rather than across the batch

D

GroupNorm, because it requires a fixed number of channels shared across the whole batch