To the first part of of your friends question I would tell them,'no if anything being armed enhances my safety and that of everyone around me. I practice situational awareness and seek to avoid conflict if at all possible, my pistol is only for use in life or death situations, not to win arguments or be a policeman. Most likely if I have to use my pistol the situation is dire and having been unarmed will not make it better. However, most criminals will avoid armed victims and in the highly unlikely event of a gun grab attempt my holster has retention capabilites and I have trained to resist such an attack.' Thats how I would answer.
I might even say that by being armed is announcing that one will defend themselves and those around them, what are the unarmed saying by being so? If your friend is attacked you can help or save him, if you are attacked your unarmed friend may not be able to do much of anything to help you.
It is also useful to bring up the issue of trust, especially with family. If anyone doesn't trust you while armed, do they really trust you unarmed?