Having some way to upload the sound data to your computer I think is a big must as well. Some recorders can only play back the sound.
Another plus, but not really a requirement would be removable storage. Many of them store the data internally and upload through USB. But some accept microSD cards, which would allow you to expand your capacity. Also, memory cards are very robust, able to withstand even a trip through your washing machine (believe me, I've done it several times lol). So even if your recording device gets destroyed, your data will still be safe on the card.