Group association based on a small group of extremists is asinine. It's the same as blaming all Americans for the acts of a soldier/group of soldiers who inappropriately tortures a POW. Unless a person or group is actively saying "yeah, terrorism is awesome and justified in the name of the sky wizard!" you should presume the group does not support such barbarism.
As for the last bit, you beg the question when you say "ideologically speaking, represents the same extremists." No, a muslim community center does not represent those extremists any more than the YMCA represents the crusaders or modern christian terrorists. It's intellectually dishonest to suggest a link or association, and pulling out the "it's insensitive" bullcrap is wearing thin. Know what's really insensitive? People like you who are calling Muslims terrorist supporters or representatives of extremists because they want to build a f'ing community center in a place they have every right and reason to. That is insensitive, not the planned Muslim community center.