0

enter image description here

This is a GRU. Now, what will be the value of $h_t$, at $t$=$0$. That is, what will be the value of the hidden state at just the starting?

Batman
  • 1

1 Answers1

0

If you think intuitively what h_t (that is h_t-1) shoud represent is, very freely speaking, amount of rembering from the previous steps. Since you are just starting, and want to remember everything, very intuitive thing is to set the value at 1. Than you can see in the equations that the mutiplication with h_t-1 makes no difference in the beginning.

Noah Weber
  • 5,609
  • 1
  • 11
  • 26
  • Thank you for your answer. Can't we initialize with 0? Would it make the network worse? Because I saw somebody initializing with zeros on the internet. – Batman Nov 01 '20 at 18:02
  • hmmm well if you insert it in equation terms will dissapear, you see that right? – Noah Weber Nov 01 '20 at 20:13