Chuck is right that there could be some galvanic action contributing to your issue. One other possibility is that your aluminum has been contaminated with steel particles at some point. Your second picture does seem to clearly show rust, which wouldn't be happening if the metal was all aluminum. It's less clear what the issue is in the first picture, but it could be smaller pieces of foreign material that interfered with the anodizing coating.
If this is the case, there are a few possible sources. The most likely is that the tooling used for the machining of your parts had been used on steel before. Since aluminum (at least the common alloys and tempers) is a very soft metal, it is easy to embed particles of other metals in the surface. For this reason, many shops will segregate cutters, grinding wheels, files, etc for ferrous and non-ferrous metals. Selection of the material used on the cutting tool, and speeds & feeds also have an impact on the risk of entrapping foreign material. Other possible sources are defects in the aluminum stock, contamination in the anodizing baths, or impact with a steel object.
Based on your picture, I would guess that steel was embedded while the part was machined, and then the moisture in the anodizing process accelerated the rusting. This is problematic not just cosmetically, but also bad for the coating because as the rust progresses, it will expand and separate the anodizing from the base aluminum. The best way to solve this problem is probably to use dedicated tooling for these parts that is only used on non-ferrous materials, so there is minimal risk of getting steel into the aluminum in the first place. This segregation is especially important where aluminum is to be welded, because a small amount of steel in the weld pool can contaminate the weld and cause excessive porosity.
Assuming this is the problem, cleaning it after the fact is fairly difficult. A chemical process won't work without damaging the whole surface, you have to actually remove the offending material or it will rust again. If it's loose enough and the anodizing layer is thin, you may be able to scrape it off with a knife or a needle file. Depending on the function of your part, you might be able to mill or face the surface down with clean tooling and re-anodize. One important thing is to take a big enough cut that you don't get any 'tearing' in the surface. Too thin a cut will tend to rub and push the material in deeper. If that's not possible, the next best thing would probably be to dig out that specific area by drilling or with a clean abrasive. Again, depending on the function of your part, you may then be stuck welding the spot full, re-surfacing, and re-anodizing.
In the end, you may decide that if the issue is only cosmetic, it's better to live with the stain than the cosmetic impact of a repair. Whatever you do, make sure you remove all of the steel, otherwise the rust stain will likely re-appear soon.
