The problem with hiding behind the permit is that it enforces the idea that you need a permit to avoid getting hassled. You shouldn't be hassled in the first place!
If a LEO stops you to harass you about your sidearm, and after discovering that you have a CHP he decides to let you be, then he hasn't learned anything. In his mind his actions were justified because he made sure that you had your permit. On the other hand, if upon being stopped by a LEO for your sidearm, you politely remind the LEO that he should mind his own business unless he has reasonable suspicion that a crime is being committed, he may just figure out that he screwed up.
I'm honestly a little ashamed that I even have my CHP at all. When I went through the process, I didn't realize that I could exercise my right without getting anyone's permission first. That's the whole point of open carry.