I can't pretend this is an easy issue, and I can't say I know exactly what the law should be (if any).
And if I saw you at Knob Creek wearing that, I would WANT to assume you were a veteran who earned what you were wearing on your fatigues. Some things are more important than others, of course. A flag may be appropriate whereas LT bars and a unit patch might not be. I don't know where to draw the line either.
I do remember when I was a naval officer that I often thought how easy it would be to impersonate one. I think it would mostly be the air of confidence and knowing a few procedures that makes the difference.