I have one of their holsters (about a year ago) that I use for OCing (in other states, not here in TX) but I forget the model #...so I went to Comp-Tac to look it up and don't see it anymore! It was an "adjustable cant" OWB holster for a 1911 -- must have been discontinued as I don't see my exact holster being offered currently.
Anyway, in answer to your question, NO, I have not had any issue of the tension-adjusting screws falling out or getting loose with my Comp-Tac kydex holster. I DO, however, need to tighten them down fairly snug to get the gun-drawing friction to my liking, but that's what the screws are for. This compresses the rubber bushings more but again, the screws do not come loose. Maybe the screw's rubber bushing on that side of your holster is missing?