Since hours have transpired without a response, I'll weigh in.
A principle, arguably the primary one, of common law self-defense is 'Be innocent of instigation.' Another is 'Attempt to withdraw.'
Were your hypothetical to escalate to violence then I would deny both the common law privilege of self-defense.
The elements of common law self-defense are four; Be innocent of instigation; Be in fear of harm; Use sufficient force only to deliver oneself from evil; Attempt to withdraw.
An armed society is a polite society.