5

I'm trying to do a simple softmax regression where I have features (2 columns) and a one hot encoded vector of labels (two categories: left = 1 and Right = 0). Do I need to standardize just the vector of features or also the vector of labels? when I do that all my zeros and ones transform in different numbers and also I don't know how to identify who is the Left or the Right category. I'm using mxnet and gluon. Here is how I standardize: labels = (labels - labels.mean()) / (labels.max() - labels.min())

labels before standardization: [0. 1. 1. 1. 1. 1.

labels after standardization: [-0.5633803 0.43661973 0.43661973 0.43661973 0.43661973 0.43661973 ...

How can I after identify (with strings) if my prediction is actually giving me Left or Right?

José Lucas Araújo dos Santos
  • 51
  • 1
  • 2

5 Answers5

5

NO, you do not standardize labels

The purpose of standardization is to bring features with disparate ranges into a standard range. When the data is not standardized, features with large numerical values will tend to have a larger influence (weight) than those that are smaller numerically.

Consider the Automobile Data Set (https://archive.ics.uci.edu/ml/datasets/automobile) from UC Irvine. Among the features, it has height, lenght, width, weight, # cylinders, along with a number of other label and numerical features. Height ranges from from 47.8 to 59.8 and weight ranges from 1488 to 4066. You'll want to standardize them, but not label features such as body-style or engine-type.

Anandologist
  • 411
  • 3
  • 5
1

From what I can tell, there isn't a "right" answer to the title question. Most people I know wouldn't bother. (Indeed, one often-used scaler puts the data into the range $[0,1]$ anyway.) https://stats.stackexchange.com/questions/290929/standardizing-dummy-variables-for-variable-importance-in-glmnet
https://stats.stackexchange.com/questions/359015/ridge-lasso-standardization-of-dummy-indicators

For the second question, normalization/standardization is always (er, always that I've seen, and certainly for your example) applying an increasing function, so that the ordering is preserved. For your normalization of binary variables then, always 0 gets mapped to the negative value and 1 to the positive one.

Ben Reiniger
  • 11,094
  • 3
  • 16
  • 53
1

While there may not be any added value, is there is any harm in applying standardization to features which were already one-hot encoded ?

thereandhere1
  • 715
  • 1
  • 7
  • 22
0

It depends on what sort of model you're training and how you're going to be consuming these one hot encoded vectors. For example, if you're going to use a deep learning based approach, you probably don't need the one-hot encoded vectors, and you can use an embedding table to project to a higher dimensional space.

If you're going to send this into a linear classifier, I'd like to think that it should not matter too much since the coefficients will be learned accordingly. Performing normalization does make the inference time a bit harder since you need to remember to apply the same normalization to ensure that the data statistics remain unchanged, so there's that.

This has also been discussed here before Should one hot vectors be scaled with numerical attributes

tl;dr try it and see the impact it has on your dataset and problem.

dhruvbird
  • 101
  • 1
0

No you don't need to.

To elaborate my answer:

  • Your one hot encoded labels has already converted into categories.
  • And you can not use, or it's useless to use standardization techniques over them.
Harshad Patil
  • 822
  • 1
  • 2
  • 13