It sounds like you are "limp wristing" and "night shooting".
If you fire a semi-auto and allow your wrists to bend at the moment of firing (Limp Wristing), the casing has a tendency to fly straight back, hitting the shooter in the head. It will also cause an excessive number of failure to eject stoppages.
If you flinch at the moment of firing, you are "Night Shooting" or firing with your eyes closed. When you close your eyes and fire, your wrists will break downwards and the round will almost alwaysimpact extremely low, sometimes hitting the ground directly in front of the shooter.
If the sights were misaligned on the slide, I would expect rounds straying left or right, not up and down.
I've carried SA .45's for years and will attest to their accuracy and reliability. My best advice for you is to seek professional shooting instruction.