Would it make sense to first train a model on images from multiple domains, and then do "fine-tuning" on one specific domain to improve its performance on it?
For instance, one could train an object detector based on cars camera recorded in NYC, Paris and Beijing, then continue training on Paris only. For a model that would be deployed on Paris only, should we favor diversity or specificity? And does this training method has a name?
Generally speaking, training with a broad distribution of data and then fine tuning on a specific domain generally leads to better performance and faster training time.
There are techniques such as meta-learning, domain invariance using techniques (ex. Gradient Reversal) or minimizing distance between the mean domain feature space using Maximum Mean Discrepency metrics. You could also familiarize yourself with empirical risk minimization and develop an intuition grounded on experiments. You could also check out 'A theory of learning from different domains' by Ben-David et al.
