Wikipedia says, "To date, no person has ever been charged with a crime for refusing to answer the ACS." The Census Bureau says that it prefers to gain willing participation. However, the law does create the possibility of a $5000 fine for not cooperating.
I suspect that no one has been charged because they fear losing their stick in a court challenge. They prefer to be able to say that there is a fine rather than try to collect it and losing the ability to say it exists.