I carry the XDM40 in 3.8" (full length grip, not the compact one) and use a SERPA with the paddle attachment, quick detach, and carry it at the 10-11 position. I find it to be comfortable, the paddle attachment makes it easier to adjust (the belt attachment will hit the belt loops), the quick detach moves it slightly away from the body (both more comfortable when bending over and less likely for my shirt to cover it since I don't tuck in my shirt) and allows me to easily remove the gun but leave it in the holster, and the 10-11 position I find to be more comfortable while also keeping the gun in a harder to grab position (same reason why I don't put a wallet in my back pocket).
When I CC it (mainly back before OK was an OC state) I would carry it in the nylon Blackhawk holster at the 10-11 position. While the clip on the holster isn't the most secure, at the position I carried it the gun was plenty secure so long as I had a belt on. When I tried it in board shorts the clip would flop around a bit and made me feel uncomfortable with it. Likewise when I carried it anywhere that wasn't in the front the nylon would flex and feel like the gun was falling away from me. But in the front with a belt the holster was comfortable and I could even run without the gun moving.
Note that for attire I generally wear casual button-up shirts (the kind that you don't tuck in) or tee shirts, shorts, and that I'm a lefty. I've found that what's comfortable and/or what works changes depending on the clothes worn.