Two major points to add here.
First, the very next sentence after the code snippet posted by Grapeshot says this:
"However, this section shall not apply to any person engaged in
excusable or justifiable self-defense."
So yes, if you are engaged in an appropriate self defense situation, you should be OK.
Second, one of the most useful resources I've found in understanding the scope of brandishing is the opinion rendered that upheld the conviction of Morris in
Morris v. Virginia. It's a fairly easy read, and not too long. Most of the opinion is regarding the possession, but there are also sections on the brandishing charge.
The gist of the brandishing portion of that case is that Mr. Morris' conviction for brandishing was upheld, even though he never actually touched his weapon (which was merely a flare gun, by the way.) His "act" was to lift up his tee-shirt, exposing the gun in his waistband, in response to a mildly confrontational conversation with a man about an offensive remark Morris made regarding the man's wife.
The point is, if you employ the fact that you possess or have easy access to a gun in an effort to alter someone's behavior, that's likely to cause the required components of the law to be reached, and I think you are likely to be brandishing, at least in Virginia.
TFred