10 / 10 Attention Internals

NTK-by-Parts

A model pretrained with RoPE at a 4K context window needs to serve 128K context without full retraining. Naive linear position interpolation scales every RoPE frequency dimension by the same context-length ratio. YaRN instead applies NTK-by-parts scaling. What does YaRN do differently, and why does that work better?

A

It removes RoPE entirely for long-context inference and replaces it with learned absolute position embeddings trained on the extended context length.

B

It interpolates the low-frequency (long-wavelength) dimensions less aggressively than the high-frequency ones, since low frequencies already generalize well and need no adjustment.

C

It treats each RoPE frequency dimension according to its wavelength relative to the original training length: high-frequency (short-wavelength) dimensions are left almost unscaled since they already generalize to unseen relative positions, while low-frequency (long-wavelength) dimensions are interpolated more aggressively since they would otherwise rotate into positions never seen in training; it also applies an attention temperature correction so post-softmax sharpness matches the original distribution.

D

It scales every RoPE dimension by the identical new-length/old-length ratio, exactly like naive linear position interpolation, and closes the resulting accuracy gap purely by fine-tuning for more steps.

E

It rescales the rotary frequency base only in the first transformer layer, leaving every downstream layer using the model's original unscaled frequencies.