What kind of holster are you using? If you have a high-rise holster then you'd look silly resting your hand on your sidearm. I have a big level 3 serpa duty holster (which is a pain to put on and take off), and I sometimes will rest my hand or elbow on the top of the holster in crowded places for additional retention. Usually my hand naturally rests right next to the holster, when I walk I sometimes hit it with my swinging arms. You eventually get better at not hitting your hand.