If it is not a school or a courthouse and is a county or state owned building I see no reason why you should not be able to carry there. Carry in local government buildings is covered by Virginia's preemption law.
I served as an election official here in Fairfax County for the November election. It was in a library and I carried the entire time without incident. Unfortunately, when I serve in March for the Republican primary, I will be in a school, so no carry for me.