1

I am trying to access the hidden layers when using TransformerEncoder and TransformerEncoderLayer. I could not find anything like that in the source code for these classes.

I am not using hugging face but I know one can get hidden_states and last_hidden_state. I am looking for something similar.

Do you know how I can access them?

Gingerbread
  • 13
  • 2

1 Answers1

0

Not exactly sure which hidden layer you are looking for, but the TransformerEncoderLayer class simply has the different layers as attributes which can easily access (e.g. self.linear1 or self.self_attn). The TransformerEncoder is simply a stack of TransformerEncoderLayer layers, which are stored in the layer attribute as a list. For each layer in the list you can then access the hidden layers as mentioned.

Oxbowerce
  • 7,077
  • 2
  • 8
  • 22
  • Thanks. What I want to access (and cannot) is the output of these layers. – Gingerbread Dec 14 '21 at 18:21
  • It depends a bit on how you are using these layer, but indeed it seems that you cannot get the output from the `TransformerEncoderLayer` from the `TransformerEncoder` directly as the forward pass only returns the final output without saving the intermediate results. What you could do is manually run the forward pass by looping over the different layers and pass the data through. – Oxbowerce Dec 14 '21 at 18:31
  • I see. Thank you. Can you please elaborate what you mean by "manually running the forward pass"? – Gingerbread Dec 14 '21 at 19:00
  • You can loop through the list and for each of the encoders run the forward pass, which would look something like this: `for layer in self.layers: data = layer(input)`. – Oxbowerce Dec 15 '21 at 08:08