Reminded of more
A couple more reasons subsequent replies have reminded me about:
Protection from pointy parts of the gun. Some guns, like a 1911 with ambidextrous safety, have parts which can dig into you if you don't have something between you and the sharp corners. I have literally had it draw blood on one occasion, wearing it all day. A good holster will prevent this.
On the same note, someone has already mentioned protecting the trigger from accidental/unintended use, but a good holster can also prevent some safeties from unintentionally disengaging, providing an additional level of security against unintentional discharge.
Protection of the finish/clothes when repeatedly practicing your draws. This only applies to some gun/holster combinations, but a Kydex holster will wear less on the finish of a gun than a leather or cloth one, or than one's clothing. Also the gun won't wear on the clothing if you're using a holster.
One substantial reason NOT to use a holster, in some few cases, is to aid in concealment. A holster really adds a lot of bulk and adds substantially to the bulge. If you really want to make sure no one sees what you have, carrying without a holster may help. This is probably one reason why criminals often don't use holsters.