That seems to be a fairly uncommon experience. (LEO's exempt) The last time I recall a citizen saying something and acting even a little alarmed was at the equipment auction in Apex a few years ago, I was registering for my bid card.
Guy from across the room: "HEY MAN WHAT DO YOU NEED THE GUN FOR?"
Me: "because my holster looks like $hit without it!"
That ended the whole scene right there, the people who were starting to take notice mostly giggled at my response, and were confident nothing was about to happen. As good as that played out at the auction it may not have worked so good at church etc. When I see someone give me the questioning look, I take a sec and think "what would I say if that person would have confronted me?" that helps me build up a arsenal of responses.