i would not confirm the presence of a gun!
florida V. peters tells us that, when an activity requires a permit to do,
then absent RAS of a crime, the permit cannot be compelled!
like you said, driving, and carrying a gun, even concealed, is not in itself a crime!
you do not need to honor a demand for a permit absent RAS!
just thank him for the consensual encounter,
tell him that you are done visiting with him,
and smile, when you tell him, "you are free to go"!