I agree that it doesn't delegitimize your insistence upon personal privacy, and in fact, I'm in your camp with that, privacy is vital. My issue with the term is that, when you think in historical terms about discrimination, it's people being forced out of places/positions/society because of who they are.

The difference, in this case is that you're not being forced out of participating in conversations because you're, let's say, black. Instead, you, on your own, are opting out of communication channels because of your own beliefs.