Question 822

Thirty-Four Gigabytes Of Nothing

The Netflix Prize training set held more than 100 million ratings from about 500,000 customers on more than 17,000 movies.

Treating it as a users-by-movies matrix of 32-bit floats, what is the fraction of filled cells, what would a dense array cost, and how should the trainer store it?

A

The density does not matter, because factorization is undefined on an incomplete matrix: the missing cells must first be imputed with each user's mean rating, after which the matrix is dense by construction.

B

About 1.2% filled. A dense array is 5×105⋅1.7×104⋅45 \times 10^5 \cdot 1.7 \times 10^4 \cdot 4 bytes = 34 GB, while storing only the observed ratings as (user, item, rating) triplets at 12 bytes each is about 1.2 GB, so the model is fit on the observed entries alone.

C

About 1.2% filled, but the dense array is still the right structure because the missing cells are zeros and a mostly-zero array compresses to roughly the size of the triplet list anyway.

D

About 12% filled, and the dense array costs about 3.4 GB, which fits in memory, so the sparsity has no practical consequence at this scale.