I'm fairly certain that all XD's field strip in virtually the same way, so I'll use mine as an example:
As you slide the magazine into the well, do you feel any extra resistance? If so, have you cleaned the mag well itself? If that isn't the problem, remove the slide and look down the mag well from the top. As you do so, depress the mag release button, and you should see the actual release mechanism in the front of the mag well, almost directly behind the trigger. The mag release mechanism is the little square piece in the picture that I have circled in red.
It sounds as though you may have a little carbon buildup there. IDK what cleaner/lube you use, but most come with a little straw that should allow you to reach that area. spray the spring and bar at the center of that little mechanism and wipe of the excess. re-assemble, and check it then.