The Late Bloomer Network
A small transformer trained on a modular arithmetic task overfits the training set almost perfectly early on, with near-random validation accuracy for thousands of steps, then validation accuracy suddenly jumps to near-100% far later in training while training accuracy stays flat at 100%. What is this phenomenon called?
Sign in to answer questions and track your progress
Sign In