When Serving Costs More Than Training
Kaplan et al. estimate a transformer's non-embedding compute at about FLOPs per token for a forward pass, with the backward pass costing roughly twice the forward pass, giving FLOPs to train parameters on tokens.
A 7B model is trained on 2T tokens and then deployed. How many tokens must it process at inference before serving compute equals training compute?
Sign in to answer questions and track your progress
Sign In