4

I would like to know how I can measure the performance of an imputation technique. I have read a lot about this. Most literature on the web are applying a classifier after the data has been completed. So this classifier will be used in order to make predictions. However, I am not interested to use such classifier to make predictions.

I would like to know how good an imputation method performs on a dataset. We can measure this for example my variance, mean, mean squared error and so on. I would like to know if there are similar techniques to measure the quality of the data. I will use the dataset for descriptive analysis(not for predictive analysis, e.g. training classifier)

Please correct me if I am thinking in the wrong context/corner. Thanks in advance.

Best Regards

user6046209
  • 41
  • 1
  • 2
  • If the purpose of the dataset is for descriptive analysis, why do you need to impute anything? Can't you just describe the data as they exist? But I presume one could compute accuracy of imputation methods by randomly removing data from a dataset, running imputation on partial data set, and then comparing the actual to computed values. – Brandon Loudermilk Mar 11 '16 at 04:59
  • I need a complete data set in order to make better and more precise estimation. But the latter, I think it might be interesting to do it in such way. Thanks! – user6046209 Mar 11 '16 at 12:43

2 Answers2

2

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does not alter your answer or at least measure some effects of the missing values. I will give some simple examples.

One widespread technique is to replace missing values with a marginal central estimator for each variable. This might work for a classifier like decision tree, if there are no missing patterns in your data. However using this imputation would alter your confidence intervals if you study linear relations between variables using regression. This is simply because it will alter sample variance.

Another much sophisticated imputation method is to use EM algorithm to fit the maximum likelihood estimator of the variance-covariance matrix. This estimator is unbiased and using this variance covariance matrix you can recover the linear model in an unbiased way. You then can go to analyze linear relations. But this works only for linear and log linear models and requires a lot of data and also requires to have missing data at random, which is not always the case.

Another one is multiple imputation. What you actually do is to draw data at random for missing values according with a supposed distribution. You do that multiple times, let's say at least 30 times. You analyze each data set with traditional tools and later you aggregate those results into a single set of results. This works well mostly when data is missing at random, but is tedious, sometimes inconsistent and duo to randomness it can produce different results. Ultimately, sometimes is very hard to find proper ways to aggregate the results, it depends on the analysis you use.

rapaio
  • 4,633
  • 20
  • 35
2

If you just want to do descriptive analysis it might be a good idea to do this without imputation at all.

A more complicated method is the following:

  1. You take the data without missing values from your whole dataset.
  2. Then in this dataset you manually delete data (trying to reproduce the overall missing data pattern)
  3. For this new dataset with missing values you have the real values behind the missing values (since you manually deleted the values)
  4. Then you do imputation on this dataset (with the method you would also use for the overall dataset)
  5. Then you compare your imputed values to the real values
  6. This gives a good estimate how far you could be off for the overall dataset
Steffen Moritz
  • 113
  • 8