Sorry to hear about that. I have gotten occasional negative comments about my OC, but knowing what I know about the culture in the Northeast, I'm sure it is nothing compared to what you have been getting.
The one positive I see is you mentioned no law enforcement interaction. I have to imagine that at least some of these misinformed antis have called the police on you or your brother/friends (it seems that the first reaction of many people to something that makes them vaguely uncomfortable is to call armed strangers to the scene ). Either your local police know the law and don't respond, or the respond and observe from a distance, correctly not making contact for perfectly lawful behavior.