Think of the self-defense exception to the brandishing statute as your get-out-of-jail-free card for the time when you've decided you need to kill another human being for good and lawful reasons, and you take all the steps necessary to accomplish that goal except pull the trigger. Then you decide your initial perception was in error, and you re-holster the gun without shooting. Point is, you're not required to kill someone even if you had the requisite "good faith belief, based on objective fact, that you or another innocent third party is faced with the imminent threat of serious bodily injury". You can, and should, avoid shooting where you can see you made a mistake that changes your perception of things (e.g., it's not a burglar, it's the Alzheimer's patient from across the street who wandered into the wrong house at 2:00 a.m. by mistake).
That's the only situation, in my mind, to which that exception applies.
If you are a cop and you have to control people in the crack house at midnight while your colleagues are executing a search warrant, then, yes, you can hold the gun on people for the purpose of intimidation (though it would be illegal to shoot them unless they did something that threatened serious bodily injury).
Any other attempt at intimidating, coercing, or controlling other people using a dangerous weapon (e.g., crowbar, tire-iron, etc.) is illegal. Notwithstanding movies to the contrary, pulling a gun out does not make you the boss, it makes you the target. You may, however, use such force as may be reasonably necessary to remove trespassers from your property, and if they resist, you may escalate as necessary; if they threaten serious bodily injury, then deadly force is legally excusable - but that's the self-defense rule at that point, not the self-help rule for dealing with trespassers.
I think it would be funny if you were to photograph these "unrulies", find out who they are, and file a civil suit against them for damages and an injunction. They wouldn't have a clue how to deal with that. And when you get your injunction awarded on your decree pro confesso (told you they wouldn't have a clue), and they continue trespassing, at that point you file a motion for a rule to show cause why they should not be held in contempt. Two or three of those on the same person can and likely will result in jail time, even though it's a civil suit.