5

This might be a weird question but I'm trying to have a deep understanding of how neural networks work theoretically.

I was doing some tests with my perceptron and I decided to test it on a single input single output dataset. What I was looking for was 100% accuracy since what I was testing on was a trivial separable dataset with binary output. However, the accuracy was below 50% (it wasn't balanced). I realized that this is due to the fact that there is a single weight that when trained is either going to be =>0 or <0. So after going through the activation function (e.g. hardlim), unless the input is scaled to be [-1,1] range, it's always going to give the same output.

Does my explanation make sense? Are there any theoretical aspects I'm missing?

Also, are neural networks useful at all for single input problems? They seems useless to me since we're basically classifying by putting a point or points (for multiclass problems) on a single line to separate outputs, which is a pretty simple problem that doesn't need the intricacy of neural networks.

MaTHStudent
  • 151
  • 2
  • 1
    Does your network have one or more hidden layers? What are their sizes? Does the output neuron in your network have bias as well as weight? A graph of input vs output for your training data might help (I am imagining it looks like samples from a step function, but not 100% sure) – Neil Slater Apr 14 '18 at 07:41
  • 1
    Give your neural networks a bit more power and then see what they can/cannot do, as Neil said, add more hidden layers, bias, Also Use a single input neuron and single output neuron without any hidden layers will definately make neural networks useless... – Aditya Apr 14 '18 at 11:35

1 Answers1

1

You have saturation of the weights which results in constant predictions.

There are several ways to fix that:

  • Replace hardlim with a standard activation function like ReLU or sigmoid
  • Pick a different random initialization
  • Normalize inputs
Brian Spiering
  • 20,142
  • 2
  • 25
  • 102