Question 924

The Critic Is Half the Memory

An RLHF run on a 7B policy uses PPO with four resident models: the policy (trained), a value model of comparable size (trained), a frozen reference model, and a frozen reward model. Trained models carry mixed-precision Adam state at 16 bytes per parameter (2 bytes fp16 weights, 2 bytes fp16 gradients, 12 bytes of fp32 optimizer state); frozen models cost 2 bytes per parameter.

GRPO drops the value model and estimates the advantage baseline from the mean and standard deviation of rewards over a group of GG sampled outputs. What does that save?

A

About 126 GB, because the reward model goes too: group-relative advantages replace scoring entirely.

B

About 112 GB, and the trade is free, since sampling GG outputs per prompt costs no additional compute.

C

About 112 GB of the roughly 252 GB: the critic was the second 16-bytes-per-parameter model, and its baseline is replaced by group-relative reward normalization, which costs extra rollouts rather than resident memory.

D

About 14 GB: only the value model's weights disappear, since its optimizer state is shared with the policy's.

E

Nothing: dropping the critic raises gradient variance, so GRPO holds a second policy copy of the same size to compensate.