6

I was just wondering what the best approach is for training a neural network (or any other machine learning algorithm) where the order of the inputs does not matter.

For example: f(x1,x2,x3,x4) = f(x2,x1,x3,x4) = f(x2,x4,x1,x3)

My current approach is to just randomize the order of every training sample (my network has 44 inputs). It kind of works but it causes the result of the validation loss to jump around every epoch a lot.

Maybe something related to embedding?

There are other questions that refer to this but with a variable number of inputs and typically in regards to a RNN. I'm talking about the simple case of a fixed number of inputs where order doesn't matter.

Thanks!

simeon
  • 173
  • 4
  • So all the inputs get treated the same for your function? Your solution _should_ work. 44 inputs _is_ quite a lot though. Imagine you want to train the network all possible inputs. If you have a sample with size 44, you could order those values in `44!` ways (if I am correct). If the order really doesn't matter, and the inputs are treated the same, maybe you could some preprocessing of the data – Thomas Wagenaar Jun 01 '17 at 09:40
  • What are you trying to model? (i.e. what "concepts" are behind your inputs and output?). This information may give clues on how to go about the layout of the inputs. With no information, I could only assume that the relative values of the inputs are the meaningful thing here, regardless of which input has which value, so I would just recommend ordering the inputs by their numerical value before feeding them to the network. – noe Jun 01 '17 at 10:05
  • I should have clarified a little. The inputs refers to a numerical skill level of players of two sports teams (22 each). Thanks for the suggestion in regards to ordering - that worked surprisingly well! – simeon Jun 01 '17 at 10:21

1 Answers1

3

The suggestion of ncasas is a good one but not very clean. This ordering makes a lot of sense when it's 1-dimensional, but when you introduce more features the ordering will become more and more arbitrary. This is a problem I have come accross multiple times. This paper (https://arxiv.org/pdf/1612.04530.pdf) tries to tackle permutation equivariance, which is not exactly the same as your problem, where you would want permutation invariance. With the equivariance the output from the layer still gets shuffled if you shuffle your inputs, but the values will be the same due to the weight sharing. You could extend this idea fairly easily by using a pooling operation (max pooling or mean pooling) on the output from this permutation layer. This means you get one number output out of your layer. Repeating this with n different weight matrices you can get n outputs, which can then be fed into a normal feed forward network. Caution: This is something I just thought of based on your question and their paper, I never tried it but I don't see why it wouldn't work. In the case of sports teams you would use two lists of inputs, one for each team. Based on what you want to predict you could even fit these permutation layers into a siamese network, introducing even more weight sharing!

Jan van der Vegt
  • 9,328
  • 34
  • 52
  • Thanks for pointing me to that paper! I will have to experiment and see how effective it is. The siamese network works well and its what I've been playing with. Its hard to tell if it works better than just sending all 44 players through a normal dense network because of the noise in the validation loss values. I believe I am at the limit of how much information can be obtained through my skill values as the accuracy metric on the training set does not increase regardless of how big I make the network. – simeon Jun 01 '17 at 12:31