You were in public and had no expectation of privacy unless the photos were for publication (and possibly not even then if you weren't the focus of the article.) Legally, you can photograph anything you can see, and it's only out of politeness that someone would comply with a request to cease and desist.
You didn't do anything wrong, but neither did the curious phonephotographer.
The only thing you might Possibly have done any better would be to have given a cheerful "Hi!" and made a new friend, or at least convinced a concerned citizen that just because someone is "different" doesn't make them "bad".