I punch metals before I start a bit in them, but with small-diameter bits I am finding that they wander inside the material. E.g., drilling a 1/8" hole in 1/2" of aluminum the (titanium-coated HSS) bit will come out the other end a measurable distance to a side (orthogonal to the drill feed direction) from where it entered.
To be clear, the bit starts where I want it to since I punched the metal. But the bit appears to be flexing within the material, causing the resulting hole to be angled.
How can I prevent this? Is there something about the material that I need to understand in order to prevent the bit from wandering and causing the hole to be skewed? I'd like to know the mechanics behind the materials that is causing these skewed holes.