Sure, in practice that's the way things will be, and I'm not prepared to argue otherwise.
But I submit that it's dangerous to justify this stuff "just because," or to ignore philosophical inconsistencies because they're inconvenient. It doesn't mean we necessarily have to rush headlong into anarchy (I wish!), either. For instance, the above can be largely resolved by using different (philosophical justification: instead of justifying authority of popular mandate "just because", we should utilize a
rigorous appeal to rights (and, I would argue, non-aggression), or at least some individual application of personal morality. Yes, this is more limiting, but that's the whole point.
Jefferson, for instance, accepted popular will as the ultimate source of political power, but he
justified it on the grounds that the people were the "safest repository" for their own rights – that no beneficent third party could truly be interested enough to do as reliable a job in preserving them (an early appearance of the
Skin in the Game theory). I actually share his reasoning for the most part, but I'm afraid it only actually
works on a relatively small scale – that in a country as huge as ours, and with a Federal government with so much legislative authority (pretty much everything I'm referring to is commerce-clause-justified crap), it's far too easy for "minorities" consisting of literally tens (potentially hundreds) of millions of people to find themselves on the short end of the legislative stick, being jailed or having their property literally stolen for harming precisely nobody.
Consider, for instance, the countless people who have had everything they own
literally stolen by the majority (we're talking explicit laws passed directly under the "legitimate authority" of your "popular will") to pay for fancy toys for a bunch of criminally aggressive drug/law enforcement agents, all for having the temerity to grow some silly damn plants, or the millions another "majority" would gladly (according, at any rate, to those who claim to represent and act under the mandate of this supposed "popular will") deprive of their firearm-related property for having the temerity to own
that.
If
that really is the will of the "majority", then the majority needs to grow the hell up before they exercise any authority whatsoever.