2

I am working on a real-time recommender system predicting a product to a user using deep learning techniques (like wide & deep learning, deep & cross-network etc). Product catalogue can be huge (1000s to 1 million) and for a given user, the model needs to be evaluated against each product in real-time. As scalability is an important concern, is there any way to reduce the serving time complexity by tuning model architecture?

Shayan Shafiq
  • 1,012
  • 4
  • 11
  • 24
Aljo Jose
  • 21
  • 2

1 Answers1

0

You write ... the model needs to be evaluated against each product in real-time., which gets me thinking that you use a binary classification (sigmoid in the final layer) architecture with negative sampling for the user/item interactions when training your model.

Have you considered using multi-class classification instead? Thus, for the user only predict once for the entire product catalogue, and selecting the top-k candidates from the softmax-layer. This way, you only need to feed-forward once through your neural net during inference.

Marcus
  • 151
  • 3
  • 1
    the network predicts click-through rate (CTR) for each product and we choose the top CTR products to show to the user. Product_id is an input to the network which makes multiple evaluations against multiple product_ids during inference. I am not sure whether softmax can handle such a scenario. – Aljo Jose Jan 12 '21 at 13:25
  • If "the network predicts click-through rate (CTR) for each product...", then you only need to feed-forward once during inference right? Is this too slow for real-time you mean? – Marcus Jan 12 '21 at 16:41
  • it is a feed-forward, but for a given request data, CTR against each item needs to be computed. if there are 10K products, I would need to prepare a matrix of 10K rows with same request data changing product ids and response needs to be in less than 100 milliseconds. – Aljo Jose Jan 12 '21 at 19:33