Reducing the dimensionality of word embeddings

Question

I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some dimensionality reduction method to scale the word embeddings with 300 dimensions down to 50 dimensions?

what method of word embedding are you using? – lollercoaster Jul 28 '15 at 20:23 — lollercoaster, Jul 28 '15 at 20:23
@lollercoaster word2vec and GloVe. – Franck Dernoncourt Jul 28 '15 at 20:26 — Franck Dernoncourt, Jul 28 '15 at 20:26

score 7 · Answer 1 · edited May 07 '23 at 18:20

7

There is a paper on this subject called

Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak

You can read it here

You can also find the implementation here

In my opinion it works quite well

edited May 07 '23 at 18:20

Franck Dernoncourt

5,573
9
40
75

answered Sep 06 '18 at 14:23

Gabriel M

171
1
7

score 4 · Answer 2 · answered Sep 21 '17 at 16:28

t-distributed stochastic neighbor embedding (t-SNE) is often used for dimensionality reduction in word embeddings. t-SNE maintains the relative relationships between the vectors.

Most often t-SNE is used for visualization, thus reducing the dimensions to 2 or 3. It could also reduce the dimensions down to 50.

Reducing the dimensionality of word embeddings

2 Answers2