Just remember, as you seem to be very aware, your appearance WILL provide a frame of reference in the minds of others, whether we like it or not. If in doubt, dress in a manner that presents you to others as YOU desire them to view you. If you are also wearing any clothing that identifies you with a specific group, others WILL view you AND that group from the image you present.
For instance, if some guy was wearing a shirt that has a VPC logo on it, and also a frilly skirt, people WILL gain a view of you and that group, whether that is the intent of that guy or not.
I personally do not go nuts over my appearance, but my typical attire when I am OC is blue jeans, pocket-t or button shirt, belt/holster, and a generally neat appearance. It is GOOD advice to dress in a manner that presents a positive vision in those you encounter. That is not to be misconstrued as any form of "dress nice or don't carry." It is just advice based upon reality.
On the other side, if you dress in a manner that presents the image of a gang-banger flashing signs, and are OC, others WILL judge you based upon that appearance.