This is a fine argument to limit the extent to which anti-discrimination infringes on private business. I like to use this argument when it comes to excessive laws imposed on landlords.
But the reason the laws are not and should not be entirely uniform is the question of where power lies.
Why do we prosecute adults for having sex with children while not prosecuting children who have sex with adults?
Why are doctors, teachers, and cops prohibited from having sex with their patients, students, and inmates, while patients, students, and inmates are not prosecuted for having sex with doctors, counselors or jailers?
The clear answer is that the law is in place to protect those in a position of weakness from those in a position of power.
Why are businesses required to meet certain building safety codes that are not imposed on residential construction?
Because businesses invite in the entire community, while a homeowner doesn't tend to do that. And relative size of buildings means far fewer people at risk at any given moment.
Businesses are presumed to be in a position of power over employees. And in most cases they are. Hence, labor unions were created, and a host of labor laws then passed incorporating most all the legit concerns of labor unions (thus leaving labor unions mostly irrelevant and having no purpose but to demand ever higher pay regardless of actual productivity).
Similarly, a business can generally survive while discriminating against a minority, while that minority might have a very difficult time accessing basic services if a business discriminates.
I recall the account of a black woman in the Jim Crow South who told about the only grocery store in town that did not have prices on any items. Customers would hold an item up and ask the owner how much it was. He would blatantly charge black customers 50% more than he charged white customers.
And to top it all off, an individual is an individual and all his individual rights must be respected.
A business owner who opens his business to the public assumes some different role than merely an individual.
The real question is why it is not ok to discriminate against racial minorities, sexual minorities, religious or political minorities, but still ok to discriminate against peaceful, lawful gun owners.