The Late Bloomer Network

A small transformer trained on a modular arithmetic task overfits the training set almost perfectly early on, with near-random validation accuracy for thousands of steps, then validation accuracy suddenly jumps to near-100% far later in training while training accuracy stays flat at 100%. What is this phenomenon called?

A

Double descent

B

Label leakage

C

Grokking

D

Catastrophic forgetting

E

Mode collapse