Carrying an unloaded pistol is just stupid in my opinion. Carrying an unloaded weapon of any kind is just like carrying an expensive rock and just as useful.
I always carry with the weapon loaded and one in the chamber. I have a Springfield XD-40 Service, so there is no manual safety unless you count the grip, trigger and pin safety. And I feel completely safe doing so.