Evals 0/1

Evals

Benchmarks

Public benchmarks such as HumanEval and SWE-bench: what each task measures, how it is scored and where a high score stops predicting real use. Labs announce models with these numbers, so reading them critically is part of choosing a model.

Reading

  1. Evaluating Large Language Models Trained on Code(opens in a new tab)

    Chen et al. 2021 60 min

    Codex, a GPT model fine-tuned on public GitHub code (a separate production version powered GitHub Copilot), and HumanEval, 164 hand-written programming problems graded by unit tests, with an unbiased estimator for pass@k.

  2. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?(opens in a new tab)

    Jimenez et al. 2024 ICLR 2024 40 min

    SWE-bench: 2,294 real GitHub issues from 12 Python repositories, where a model must edit the codebase so that the tests from the fixing pull request pass. The best model at publication resolved under 2 percent.

  3. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models(opens in a new tab)

    Thakur et al. 2021 NeurIPS 2021 Datasets and Benchmarks 45 min

    BEIR: 18 retrieval datasets from different domains for testing retrievers on data unlike their training data. BM25 proved a strong baseline, rerankers and late-interaction models scored best at high cost, and dense retrievers often fell behind.