1

Below is one example Attention-based Encoder-decoder network for multivariate time series forecasting task. I want to visualize the attention weights.

input_ = Input(shape=(TIME_STEPS,N))
x = attention_block(input_)
x = LSTM(512, return_sequences=True)(x)
x = LSTM(512)(x)
x = RepeatVector(n_future)(x)
x = LSTM(128, activation='relu', return_sequences=True)(x)
x = TimeDistributed(Dense(128, activation='relu'))(x)
x = Dense(1)(x)
model = Model(input_,x)
model.compile(loss="mean_squared_error",optimizer="adam",metrics=["acc"])
print(model.summary())

Here is the implementation of my attention block:

def attention_block(inputs):
    x=Permute((2,1))(inputs)
    x=Dense(TIME_STEPS,activation="softmax")(x)
    x=Permute((2,1),name="attention_prob")(x)
    x=multiply([inputs,x])
return x

I will highly appreciate if a fresh implementation of the attention model is provided.

Debashis Sahoo
  • 113
  • 5

1 Answers1

1

A reference: https://github.com/zhaocq-nlp/Attention-Visualization/blob/master/exec/plot_heatmap.py. It plots attentions based on matplotlib

Rancho Xia
  • 11
  • 2
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - [From Review](/review/late-answers/80026) – Ethan Jun 24 '22 at 18:31
  • I need the attention values, i.e. the probability values, and if you have any example of time series forecasting, that helps. Thank you. – Debashis Sahoo Jul 28 '22 at 11:16