Evaluating Large Language Models Trained on Code(opens in a new tab)
Codex, a GPT model fine-tuned on public GitHub code (a separate production version powered GitHub Copilot), and HumanEval, 164 hand-written programming problems graded by unit tests, with an unbiased estimator for pass@k.