Only becomes an issue if you draw your weapon.
This is true to some extent. I've heard that it is still considered brandishing even if you were to remove your outter jacket to reveal that you are carrying. The law cites brandishing as:
Virginia Code ﾂｧ 18.2-282
" A. It shall be unlawful for any person to point, hold or brandish any firearm or any air or gas operated weapon or any object similar in appearance, whether capable of being fired or not, in such manner as to reasonably induce fear in the mind of another or hold a firearm or any air or gas operated weapon in a public place in such a manner as to reasonably induce fear in the mind of another of being shot or injured."
The question then becomes, did you have a reasonable explanation to "take off your jacket" or whatever article of clothing that was concealing your firearm. If you are in an altercation in the middle of winter and you take off your overcoat to reveal that you have a gun, you might have some issues when the police come.
Same applies in armed robbery. If someone walks up to you and lifts up his shirt to reveal he's ghetto packing a gun in his waistband, he has induced fear.
In short, brandishing does not simply apply to actually just drawing the gun. :P