Scenario: I develop a model which forecasts the likely sales success of a particular enquiry based on outcomes of past similar enquiries. I then assign this likelihood score to new enquiries when they come in. The sales team use this to optimise their behaviour, giving more focus to the higher scoring enquiries, and thus imrpoving their outcomes.
When newer records are incorproated into the model, they are skewed, due to the bias introduced from having the additional score information. Over time, this causes the dataset to polarise, with higher scoring items getting higher, and lower scoring items get lower. Eventually it renders the model unusable, as more and more records fail to make the grade and end up falling to the low end.
a) does this effect have a proper name?
Edit: I believe this is a Degenerate Feedback Loop.
b) what are the standard approaches for dealing with it?
