How to Scale Your Model(opens in a new tab)
An online book from Google DeepMind on how language models run on TPUs and GPUs: rooflines, how matrix multiplications are sharded across chips and what the communication costs, and how to split training and inference with data, tensor and other kinds of parallelism, with worked examples for Llama 3.