There are a good number of what many would consider heinous behaviors that are not crimes. Even if our current system of justice worked perfectly, we would still be left with a basket of people who no one wanted to be associated with, but whom had, legally at least, "done nothing wrong."
"freedom of association" and "freedom of speech" are governmental concepts, used to limit the behaviors of governments.
They are not some core, universal rights that every individual must respect when interacting with other individual.
The accused in this case absolutely still has the citizen rights of association and speech. He can gather with people and he can publish his thoughts. The fact that a bunch of individuals have decided they don't want to gather with him is in no way a reduction of his rights.
