The current topic is derived from this topic: https://community.rstudio.com/t/39722 (optional to read).
I have the following dataset: myds :
str(myds)
'data.frame': 841500 obs. of 30 variables:
$ score : num 0 0 0 0 0 0 0 0 0 0 ...
$ amount_sms_received : int 0 0 0 0 0 0 3 0 0 3 ...
$ amount_emails_received : int 3 36 3 12 0 63 9 6 6 3 ...
$ distance_from_server : int 17 17 7 7 7 14 10 7 34 10 ...
$ age : int 17 44 16 16 30 29 26 18 19 43 ...
$ points_earned : int 929 655 286 357 571 833 476 414 726 857 ...
$ registrationYYYY : Factor w/ 2 levels ...
$ registrationDateMM : Factor w/ 9 levels ...
$ registrationDateDD : Factor w/ 31 levels ...
$ registrationDateHH : Factor w/ 24 levels ...
$ registrationDateWeekDay : Factor w/ 7 levels ...
$ catVar_05 : Factor w/ 2 levels ...
$ catVar_06 : Factor w/ 140 levels ...
$ catVar_07 : Factor w/ 21 levels ...
$ catVar_08 : Factor w/ 1582 levels ...
$ catVar_09 : Factor w/ 70 levels ...
$ catVar_10 : Factor w/ 755 levels ...
$ catVar_11 : Factor w/ 23 levels ...
$ catVar_12 : Factor w/ 129 levels ...
$ catVar_13 : Factor w/ 15 levels ...
$ city : Factor w/ 22750 levels ...
$ state : Factor w/ 55 levels ...
$ zip : Factor w/ 26659 levels ...
$ catVar_17 : Factor w/ 2 levels ...
$ catVar_18 : Factor w/ 2 levels ...
$ catVar_19 : Factor w/ 3 levels ...
$ catVar_20 : Factor w/ 6 levels ...
$ catVar_21 : Factor w/ 2 levels ...
$ catVar_22 : Factor w/ 4 levels ...
$ catVar_23 : Factor w/ 5 levels ...
Now, let’s do an experiment, let’s take the discrete variable: catVar_08 and let’s count for each of its values, on how many observations that value shows up. The table below will be sorted in decreasing order by the amount of observations:
varname = "catVar_08"
counts = get_discrete_category_counts(myds, varname)
counts
## # A tibble: 1,571 x 2
## # Groups: catVar_08 [1,571]
## catVar_08 count
## <chr> <int>
## 1 catVar_08_value_415 83537
## 2 catVar_08_value_463 68244
## 3 catVar_08_value_179 65414
## 4 catVar_08_value_526 59172
## 5 catVar_08_value_195 49275
## 6 catVar_08_value_938 26834
## 7 catVar_08_value_1142 25351
## 8 catVar_08_value_1323 23794
## 9 catVar_08_value_1253 18715
## 10 catVar_08_value_1268 18379
## # ... with 1,561 more rows
Let’s check the counts above more deeply. In order to do that, let’s do some plots on different ranges:
plot_discrete_category_counts(myds, varname, 1, 500)
plot_discrete_category_counts(myds, varname, 501, 1000)
plot_discrete_category_counts(myds, varname, 1001)
As we can see on the second plot above, if we keep on our analysis the first 500 categories with higher amount of observations and remove the rest (1571 - 500 = 1071) by replacing those values to a generic value like: Not Specified , then we are going to be using categories that have been observed more than around 40 times.
Another very good advantage of doing such level reduction is that the computational effort when training our model will be considerably less which is very important.
Bear in mind that this dataset have: 841500 observations.
Let’s do another similar experiment with another discrete variable: zip :
varname = "zip"
counts = get_discrete_category_counts(myds, varname)
counts
## # A tibble: 26,458 x 2
## # Groups: zip [26,458]
## zip count
## <chr> <int>
## 1 zip_value_18847 428
## 2 zip_value_18895 425
## 3 zip_value_25102 425
## 4 zip_value_2986 422
## 5 zip_value_1842 414
## 6 zip_value_25718 410
## 7 zip_value_3371 397
## 8 zip_value_11638 395
## 9 zip_value_4761 394
## 10 zip_value_6746 391
## # ... with 26,448 more rows
Let’s check the counts above more deeply. In order to do that, let’s do some plots on different ranges:
plot_discrete_category_counts(myds, varname, 1, 1000)
plot_discrete_category_counts(myds, varname, 1001, 2500)
plot_discrete_category_counts(myds, varname, 2501)
On this case, if we keep on our analysis the first 2500 categories with higher amount of observations and remove the rest (26458 - 2500 = 23958) by replacing those values to a generic value like: Not Specified , then we are going to be using categories that have been observed more than around 100 times.
Another very good advantage of doing such level reduction is that the computational effort when training our model will be considerably less which is very important.
Bear in mind that this dataset have: 841500 observations.
Highlight:
On the second case, we were analyzing the zip code which is a very common discrete variable. So, I would say this is a common use case.
My Question:
What would be a good threshold number or formula X , such that when for a given discrete variable, we have categories/values that have been seen less than X times on the dataset observations, it worth to remove them?
For example, probably if for catVar_08 above we have some categories/values that have been seen 5 times, probably we should not take them into consideration and replace that value by Not Specified because most likely that the machine learning model will not have the ability to learn about those categories/values with few observations.
One possible signature for the formula would be:
num_rows_dataset = <available above>
num_cols_dataset = <available above>
discrete_category_counts = <available above>
arg_4 = ...
arg_5 = ...
...
get_threshold_category_observations = function(
num_rows_dataset,
num_cols_dataset,
discrete_category_counts,
arg_4,
arg_5,
...
) {
threshold = ...
return (threshold)
}
Any idea/recommendation about this?
