2

Is someone familiar with such an approach:

Suppose I want to build a bayesian neural network, with distributions over my parameters instead of point estimates. First I train my network with standard backprop. After training I start some MCMC sampling over my network weights in order to estimate the distributions.

Somehow this approach seems very natural to me and probably there are already some papers or drawbacks, which I do not see. I am aware that sampling can be a slow process, but with a good starting point it should run quite fast. Also sampling could be run in batchmode in order to speed up.

Andreas Look
  • 921
  • 5
  • 14
  • What do you mean by training the network with "standard" backprop? Do you mean to obtain a set of deterministic weights for the network? If so, how would you do MCMC sampling? What distribution are you sampling from? – KRL Jun 27 '19 at 18:23
  • Exactly, I end up with a determnistic set of weights. Basically you can not sample from a deterministic set. A workaround would be to do MCMC sampling, since the weights should be non deterministic as by nature. I use my trained network as a starting point (max. likelihood estimate) in oder to get my MAP distribution. I could simply use Hastings Sampling, since it can transform also point estimates in distributions. I only need an evaluation function, like Cross-Entropy or whatever. In the meantime I found some work doing this approach: Neil Radford: Bayesian Learning for Neural Networks,1996 – Andreas Look Jun 28 '19 at 06:53

1 Answers1

1

There are in fact a bunch of papers on this topic. I'd recommend this one and this one. Not sure what you want to do with your Bayesian Neural Network, but maybe this one is useful too.

matthiaw91
  • 1,495
  • 5
  • 17