Why does only Adam optimiser perform well in my case?

Question

I am trying to classify texts into categories (one text can have multiple categories), to do this I used one-hot encoded labels (with sklearn.preprocessing.MultiLbaleBinarizer()). My texts are also one-hot encoded but with keras.preprocessing.text.one_hot.

This is my code :

csv_data_name = 'learning_base'
df = pd.read_csv('../../datasets/csv/' + csv_data_name + '.csv ')
df['label_list'] = np.array(df.labels.str.split('|'))
df = df.drop(columns=['labels'])


# Remove incomplete lines (lines without labels)
df = df.dropna()

encoder = MultiLabelBinarizer()
labels_binary = encoder.fit_transform(df.label_list)
output_dim = labels_binary.shape[1]

vocab_size = 200000
encoded_docs = [one_hot(d, vocab_size) for d in df.text]
max_length = 200
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

# Data division
training_set_data, testing_set_data_whole, training_set_target, testing_set_target_whole = train_test_split(
    padded_docs,
    labels_binary,
    test_size=0.1,
    random_state=0,
    shuffle=False)


model = Sequential()
model.add(Embedding(vocab_size, 200, input_length=max_length))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(output_dim, activation='sigmoid'))

model.compile(optimizer=TFOptimizer(tf.train.AdamOptimizer()), loss='mean_squared_error',
              metrics=['acc'])

model.fit(training_set_data, training_set_target, epochs=50, verbose=True, batch_size=200,
          callbacks=[EarlyStopping(monitor='acc', patience=3, verbose=True), tensorboard])

Everything works fine except when I try another optimizer the learning seems incorrect, only Adam seems to work as intended.

This is my results :

|Optimizer|Time by epoch|Strict classification succes|Precision|Recall|epoch|
|-------- | ----------- |--------------------------- | ------- |------|---|
|GradientDescent                          |00:21|0%|12.5%|0%|3 (ES)|
|GradientDescent (`learning_rate = 0.01`) |00:21|0%|3.9%|16.1%|7 (ES)|
|Adadelta                                 |00:24|0%|3.1%|25.7%|3 (ES)|
|Adagrad                                  |00:26|0%|/|/|13 (ES)|
|Ftrl                                     |00:27|0%|/|/|8 (ES)|
|Momentum (`momentum = 0.01`)             |00:22|0%|/|/|18 (ES)|
|Adam (`learning_rate = 0.01` : default)  |02:37|53%|80%|60.54%|13 (ES)|

And here is the tensorboard for accuracy :

score 0 · Answer 1 · answered Sep 20 '19 at 18:32

There could be a couple of possible reasons:

One reason could be the Adam optimizer is a combination of several other optimization techniques (e.g., momentum and running average of gradient squares). The combination of those techniques works well on multi-label text classification.
Another reason could be that multi-label text classification is a sparse problem. The Adam optimizer in TensorFlow has a sparse implementation.

score 0 · Answer 2 · answered Sep 20 '19 at 19:15

The reason may be because the fixed learning rate you chose does not fit well with the data. If you try the Backtracking version, where learning rates are adapted at every step, and where convergence can be proven rigidly for many functions, things can be better. You can look at the paper in my answer in this link:

Does gradient descent always converge to an optimum?

Also, did you try NAG? In the paper I mentioned, we see from experiments that the Backtracking NAG works quite well and stable.

score 0 · Answer 3 · answered Oct 21 '19 at 11:05

It probably doesn't. What you are comparing is not exactly fair. If you want to do it right, you should try to find the best hyper-parameters for all optimizers and compare their respective performance. What this tells you now is basically "which optimizer works best with the standard hyper-parameters and some arbitrary ones i chose".

Why does only Adam optimiser perform well in my case?

3 Answers3