1

In the field of machine learning, I'm wondering about the interest of applying feature selection techniques.

I mean, I often read articles or lectures speaking about how to reduce the number of feature (dimensionality reduction, PCA), how to select the best features (feature selection etc).

I'm not sure of the main purpose of this:

  • Does feature reduction techniques always improve accuracy of the learned model?
  • Or is it just a computational cost purpose?

I would like to understand when it is necessary to reduce the number of features and when it is not, in order to improve interpretability or accuracy. Thanks!

Sylvain M
  • 13
  • 2
  • See some more explanations here http://datascience.stackexchange.com/questions/16062/is-feature-selection-necessary/16064#16064 – DaL Mar 05 '17 at 06:56

2 Answers2

3

Feature Selection (FS) methods are focused on specializing the data as much as possible to find accurate models for your problem. Some of the main issues that drive the need for FS are:

  • Curse of dimensionality: Most algorithms suffer to grasp relevant chacteristics of data for a specific prediction task, when the number of dimensions (features) of the data is high, and the number of examples is not sufficiently big. Check here some more detailed explanation
  • Correlation between variables: Typically, the presence of highly correlated pair of variables can cause ML algorithms to pay to much attention to a particular effect that is "over-represented". For this reason many FS methods address the reduction of this correlation. Reducing the number of correlated variables oftenly increases the model predictive power.
  • Latent features: Although specific variables might be highly expressive for your problem, a lot of power can be achieved when finding "latent features", such as linear and non-linear combinations of the original variables. Here there are hundreds of approaches, from PCA to Neural Networks. Independently from the approach (and statistical assumptions) the idea is to create new features that condense the information of a bigger set of features into a smaller one. Hopefuly the new set of features is more representative, and being smaller could be more easily learnt.

Feature selection does not necessarily improve the predictive quality of the model. Reducing or transforming the features might lead you to loss of information and then a less accurate model. Is an open and complex field of research. However in many cases it becomes quite useful. It will depend on how good and different are your original features for describing the target variable. If you look into bionformatics, you'll see people dealing with thousands or even millions of features, while having only hundred of examples. Here feature selection becomes increasingly relevant.

PS: Most commonly I've seen the term "feature selection" used for the creation of compound features, as most examples I've mentioned, while Feature Extraction term is used for actually removing specific features from the dataset without taking into account is relation with the target variable.

Javierfdr
  • 1,490
  • 12
  • 14
-1

Feature selection could be done to prevent overfitting. The more features you have, the more likely you'll overfit. And yes, I'd say one of the purposes could be to reduce computational costs.

User1915
  • 109
  • 2
  • Okay but reducing the feature space (so regarding to what you've said, regularizing) may lead to a worse model? Is there a metric that could help us deciding if we should reduce or not the features ? – Sylvain M Feb 28 '17 at 09:40
  • @SylvainM it does not necessarily lead to a worse model. For example, you have many features, but some of them are garbage features. Then by removing those garbage features, you can actually improve model accuracy. This improvement depends on how well you have chosen the features. Usually, I'd say if you are overfitting, then definitely try some feature reduction. – User1915 Feb 28 '17 at 10:12