Brandishing is displaying in a menacing or threatening manner.
"Definition of BRANDISH
1: to shake or wave (as a weapon) menacingly
2: to exhibit in an ostentatious or aggressive manner "
From your description you were not doing this. Even if you unholster and show your weapon to a friend, in public. (although not recommended, only because we don't want to alarm people unnessecarily), it is not brandishing.
People really get freaked out about seeing a firearm, even more so when they see it out of a holster. But come on people, there are times when you have to remove a holstered weapon and store it, whether it's still in the holster or not.
Do what you need to do to secure yourself and your firearm, without doing it in a threatening manner, and you'll be fine. Just know, that a holster is not required to open carry, unless it is a requirement of the state. Just because I've never seen it, doesn't mean some state doesn't require it.
Of course there are always people that will tell you otherwise, because they aren't informed.