0

I am currently working on a Transformer architecture. Trying to picture an RNN (or Encoder) as a normal Feed Forward network really confused me after looking at the following image in an article:

(Image 1) Image 1 - Encoder architecture

I am usually used to seeing it like this:

(Image 2)

Image 2 - Unfolded Encoder architecutre

In Image 1, it shows that the input goes in ALL at once, whereas in Image 2, we see only a single input at a timestep.

My two questions:

  1. Does this mean that Image 1 is a single node in Image 2?

  2. How can I picture an RNN-architecture as Image 1?

Fishie
  • 1

1 Answers1

1

I cannot be sure whether this graphical representation is fully correct or wrong, but your misunderstanding looks like it revolves around what is an LSTM cell and LSTM neurons.

In the second figure, the nodes depicted should be LSTM cells. A cell is in essence a forward neural network consisting of neurons and so an analytic representation of it may look like what the first figure shows, although admittedly it looks very confusing.

See a recent discussion on that exact matter here What is the difference between a "cell" and a "layer" within neural networks?.

Figure 1

hH1sG0n3
  • 1,978
  • 7
  • 27
  • So the question is not regarding the LSTM cell. It is about how we can depict Image 2 (which is highly abstracted) as Image 1 (to show all the nodes and the weights connecting to the nodes in each layer). I should have phrased the question a bit better. – Fishie Aug 27 '20 at 00:25