10

I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf

In order to select which CF algorithm should be used the paper refers to the density of the dataset. What it doesn't do is explain how you actually calculate the density of your dataset.

So in the context of that above paper can anyone help explain to me how I would calculate the density of a dataset? The paper refers regularly density in the 1-5% range.

djones
  • 203
  • 2
  • 4

1 Answers1

7

It's actually defined on the first page:

... sparsity level (ratio of observed to total ratings) ...

In other words, the fraction of the user/item rating matrix that is not empty. Remember that the problem is that most user-item pairs have no rating, and we wish to estimate them.

Example:

Let there be three users and four products. The number of possible ratings is $3\times4 = 12$. If every user rates only one product each (regardless of which product), the density is 3/12 = 25%.

Emre
  • 10,481
  • 1
  • 29
  • 39
  • 1
    so given this simple example data: User 1 rates Product A. User 2 rates Product B. User 3 rates Product A. Product C and Product D exits with no ratings. What would the density be? 50%? – djones Mar 07 '16 at 21:35
  • 1
    There are three users (1,2,3) and four products (A,B,C,D). There are three ratings (1A, 2B, 3A), hence the density is 3/12 = 25%. – Emre Mar 07 '16 at 21:48
  • 1
    @Emre You should put that example into your answer. – Martin Thoma Mar 08 '16 at 18:15
  • I really like the answer here, but. To be more precise, sparsity and density are fraction of non-filled ratings and filled ratings respectively. They can not be treated interchangeably. Though, density + sparsity should result in 1.0. I know that due to definition from the article @Emre gave the good answer. But the definition of sparsity as actual density is misleading. – Bartłomiej Twardowski Mar 10 '16 at 10:48
  • I agree, but that's how they defined it, and it's common. – Emre Mar 10 '16 at 16:17