I disagree. It's not the name itself that is derogatory; it's the manner in which it is meant. And I'm confident that most people who use the name "Wally World" are using it as a moniker, not a slur.
Hence why I don't use the term. It's far too benign. I'll stick with referring to them as "The Evil W" and shop elsewhere, no matter how cheap their ammo is.
Though, I have considered only shopping them for their "loss leaders". I.e. things they intentionally lose money on to get people into the store. If ammo is in that category and they're losing money on it, I may just buy it there to supplement reloading.