Common sense would dictate that there is nothing wrong by wearing an empty holster, but you will be in NJ. I am almost cetain that you will be stopped, probably handcuffed, and arrested. If convicted, you will be able to start appealing this decision all the way to the Supreme Court which could take 5 or 6 years and cost you a ton of money. My honest opinion is that the US Supreme Court would decide in your favor, but the bottom line question is : WOULD IT BE WORTH IT?
It is probably best if you did not wear an empty holster while in NJ. Although this would take a lot of time, an organized event in which 100 people would show up to verbally protest NJ's draconian gun laws might accomplish more, although I am not even cetrain of that.
I wouldn't give up completely your idea of wearing an empty holster, but it would have to be thought out completely, planned to perfection, maybe even discussed with the Police beforehand about what you want to do, and you should be joined by others, not do it yourself.