Speaking in general terms (ie. without respect to the various exemptions provided for in 2C:39-6), you need a permit to carry a handgun in order to legally possess a handgun (2C:39-5b). You need a Firearms ID card to legally possess a rifle or shotgun (2C:39-5c). You may never possess a loaded rifle or shotgun (2C:39-5c(2)).
So, according to the law, you're legal to open or conceal carry a rifle or shotgun, so long as it is unloaded, and you are in possession of a Firearms ID card.
In practical terms, walking down main street with a slung rifle is going to get you arrested.