Right handed people have the same problem. Only, the sides are reversed.
I wouldn't worry about it.
Reaching near a gun is not reaching for a gun. Anybody who can't tell the difference should maybe not be allowed in public.
Think about it. You reach in the same general direction as your gun. But, by the time an observer could overreact, your hand has already passed the gun and arrived at your wallet pocket or keys pocket.
The one exception would be a cop during a detention. I would move just a little slowly. Don't wanta get shot by the partner behind me who has already illegally drawn his gun when there was no need, and over reacts by not noticing you didn't actually draw your gun.