The problem is that once you have that loophole in the ideals of free speech, people start calling everyone they disagree with "intolerant" in order to feel good about themselves while shutting down the speech of people who disagree with them.