22

I know that there is no a clear answer for this question, but let's suppose that I have a huge neural network, with a lot of data and I want to add a new feature in input. The "best" way would be to test the network with the new feature and see the results, but is there a method to test if the feature IS UNLIKELY helpful? Like correlation measures etc?

Zephyr
  • 997
  • 4
  • 10
  • 20
marcodena
  • 1,667
  • 4
  • 14
  • 17
  • 1
    A non-random correlation might be an indicator that the feature *is* useful. But I'm not so sure about pre-training tests that could rule ideas out. The paper you link makes it clear that non-linear correlations are not well detected by the available tests, but a neural net has a chance of finding and using them. – Neil Slater Jul 10 '14 at 11:28

2 Answers2

20

A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information. A low correlation between the new feature and existing features is likely preferable.

A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

If the new feature was manually constructed from a combination of existing features, consider leaving it out. The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers. Whenever possible, prefer learning features to engineering them.

Madison May
  • 2,029
  • 2
  • 17
  • 18
  • I always thought to compare the value to predict with the features, you are talking about correlation between features. Is your answer applicable also to my case? in theory I should add only new features that are correlated to the value to predict, right? – marcodena Jul 10 '14 at 19:06
  • That's also a valuable metric -- just updated my answer to address that as well. – Madison May Jul 10 '14 at 19:18
  • In short, strong correlations with the value to predict is a great sign, but weak correlation with the value to predict is not necessarily a bad sign. – Madison May Jul 10 '14 at 19:19
  • Thanks. I'm writing a report and I wanted to show the linear/non-linear correlations in order to justify the features (even before the results). Does it make any sense? From your answer I could make a matrix of correlations but maybe it's nosense – marcodena Jul 10 '14 at 19:27
  • A matrix of correlations would make more sense if you were performing linear regression, but it could also be a useful metric for neural nets. Give it a go and see what you get! – Madison May Jul 10 '14 at 21:29
  • 1
    I would use non-linear correlations, but ok thanks – marcodena Jul 10 '14 at 23:17
1

If you are using scikit-learn, there is a good function available called model.feature_importances_.

Give it a try with your model/new feature and see if it helps. Also look here and here for examples.

Ethan
  • 1,625
  • 8
  • 23
  • 39
Aniket
  • 154
  • 4