I wouldn't deem it irresponsible.
I would say it is irresponsible of the people that would grab the gun from you. I do not care how fast you are with a serpa or whatever, I will still be faster with my fobus no retention. And yes that does leave me a little bit more prone to gun grabs, but I think if you are attentive enough, you shouldnt have a problem. If someone comes up to you with a firearm in your face off gaurd, they will get your gun if you have it ziptied to your belt....
As others have point out, my holster as well comes up high enough that I can use my elbow to get leverage on it, if I need to sign something, or bend over, or hell, if I am just tired and want an arm rest
I think some people thought legitimately believe that nobody will confront them because they have a firearm, but the truth is that many people are bold enough to do this, and if you let your guard down because of the false security, and you don't have retention, by the time you notice someone pulling on it, it will be gone.