One feature that would be valuable, although I have not seen it offered on any recorders in my limited search, would be a continuous running looped overwrite mode.
You would turn your recorder on in this mode when you entered some public environment, and the recorder would start recording. It would maintain a pre-set loop period of time, say 20 minutes, recorded, overwriting the front (older than the 20 minutes) as it went along.
Until you pushed a magic button which would then enter a "normal" mode, starting at that point adding on to the previously recorded 20 minutes, until you manually stopped the recording.
This feature would give you up to 20 minutes after the actual beginning of an "encounter" to "start" the recording. No more fumbling around when you suspect you may or may not need to record an incident. This would allow the opportunity for a much more discrete activation, that is not time critical to when the most stressful conversations (the first ones) are taking place.