Scaring people should never be the intention of carrying a gun. Protection and education of civil rights are why I carry. It is very easy to get into the Billy Badass/Marshal Dillon mindset, but we are neither and have to do our best to be as polite and friendly as possible because people will be watching us.
I'm sure you were very well behaved, but if we think we are scaring people, especially four grown men, we are probably doing something wrong.
Hope you take this as intended, not a criticism but wisdom learned from many mistakes.