Counting Covariance Parameters

$K = 3$ classes, $d = 50$ features, 40 training rows per class. LDA pools one covariance matrix across the classes; QDA fits a separate $\Sigma_k$ per class (scikit-learn describes LDA as the special case $\Sigma_k = \Sigma$ for all $k$ ).

How many free covariance parameters does each estimate, and what goes wrong for QDA at this sample size?

LDA 1,275 and QDA 3,825, and QDA is fine here because the estimator regularizes the covariance automatically; the only cost is compute.

LDA 1,275 and QDA 3,825. Each class covariance comes from 40 rows, so its rank is at most 39 in 50 dimensions: singular, and QDA needs its inverse.

LDA 2,500 and QDA 7,500, since a $d \times d$ covariance matrix has one free entry per position in the grid, so $d^2$ of them.

LDA 50 and QDA 150: a covariance matrix is pinned down by the per-feature variances, so there are only $d$ numbers to estimate per matrix.