The holster that comes with the XDs is WEAK. That is, it does not protect the gun well, and arguably has worse retention than similarly designed leather holster. Get a retention holster. When I had XDs (before the canoe accident) I carried them in a Serpa CQC. Removing the firearm became second nature with minimal practice; I never felt like getting my gun out fast enough would be my main concern in a bad situation.
+1 for you SA, though.