Less dense air tends to rise; in general, how much the atmosphere cools with elevation (in addition to humidity differences) drives a difference in density, and if that difference is larger than the stable condition, air masses tend to move up. Thus afternoon thunderstorms, for example.
An inversion is the opposite. When the air higher up is warmer, it's also less dense, so the air lower down doesn't move up - the atmosphere is "extra stable". Since the air isn't moving up, air pollution isn't carried up and out by advection, and therefore accumulates as diffusion is much slower. The same mechanism that carries moisture up to create afternoon thunderstorms isn't able to move pollution out.