I almost feel your pain. I'm about 5'9" and 164 lbs. If the gun sits the wrong way it will rub bone, but I can usually get it in a pretty comfortable position by pushing more towards 8 o'clock (your 4 o'clock).
When I get the money I want to invest in a good leather holster, possibly pancake style. I think it'd sit much more comfortably, not to mention look better. But for now the Serpa will have to do for me.