0

My question regards this image:

enter image description here

It seems that after the multi head attention there is a linear layer as they mention also from here: enter image description here

the linearity is given by the weights W^{o}. my quesion is: for the decoder, doesn't this linear layer mess up with the masking of the attention? I mean, if each token from the attention layer should not depend from the next tokens then these weights seem to mess up the whole thing or they don't? Indeed these linear weights will learn the dependencies among all the tokens and during inference there could be a problem, maybe(?). Thanks for any clarification.

erre4
  • 95
  • 7

1 Answers1

1

No, this is not a problem.

If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this:

enter image description here

There, you can see how the masking of the current and future positions happens inside the scaled dot product attention, which happens before the multiplication by $W^O$. Therefore, the values learned for $W^O$ are trained only with the information from the previous tokens in the decoder.

noe
  • 22,074
  • 1
  • 43
  • 70