LoRA: Low-Rank Adaptation of Large Language Models(opens in a new tab)
LoRA: freeze the pre-trained weights and train two small low-rank matrices per layer whose product is added to a frozen weight matrix, which for GPT-3 175B cut trainable parameters by 10,000 times and GPU memory by 3 times.