6

I am using Keras for a project. I would like to know if it makes any sense to add any kind of regularization components such as kernel, bias or activity regularization in convolutional layers i.e Conv2D in Keras.

If yes, then which regularization is most useful for conv2d layers

  • Kernel
  • Bias
  • Activity

As explained here the regularization techniques are useful for the fully connected(dense) layers. Any such intuition/logic for conv2D?

Arka Mallick
  • 560
  • 2
  • 7
  • 16
  • For dear editors, now is the question enough specific? Or at least this is the question I had in mind. For the downvoters, its customary to leave a comment for down-voting, unless you are not sure why are you downvoting yourself!! – Arka Mallick Nov 20 '18 at 17:24

1 Answers1

2

Nowadays, people don't tend to add much regularisation like $L_2$ to convolutional networks. Usually, regularisation is achieved through the use of techniques like dropout and batch normalisation (although exactly how this regularisation actually happens is not well understood).

Having said that, it may be different for your particular problem. I recommend that after experimenting with dropout and batch norm, you should experiment with $L_2$ and see how the predictive performance is affected on a validation set.

timleathart
  • 3,900
  • 20
  • 35
  • Ok thanks for the comment. I get your point. I have done evaluating dropout and batch norm. Keras batch norm somehow gives me worse result than without. So I am thinking to evaluate l1/l2 regularization too and see how it goes. Having said that, I do not have idea which regularizer to use, kernel, bias or activity ?? or all of them. Any thoughts on this ? Thank you for your time and the edit :) Cheers. – Arka Mallick Nov 20 '18 at 00:30
  • 1
    I think I would try kernel or kernel+bias first. Just give different combinations a try and see what happens :) – timleathart Nov 20 '18 at 00:52