How can I apply similarity algorithm (or comparison) of over one million vectors with another one million vectors?
I am following this pyimage search tutorial but don't know how to scale up the algorithm when I have millions of images to compare. Like comparing similarity of over 1 million images with other 1 million images. It stops with an out of memory error at 100000.
You might want to take a look at this project FAISS.
Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.
It has nice wrappers for you to use from Python. Check the wiki for examples on how you can integrate this in your application.
It is built with scalibilty in mind, supporting indexing and comparisons of millions of vectors and it can be run either on CPU or GPU.
You could probably use Dask to do this.
Dask natively scales Python. Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love
Dask will schedule your computation so that you don't run out of memory and then provide you with the result of your computation. It supports many of the libraries in the data science stack including scikit-learn, pandas and numpy.
