Which is exactly why the Fediverse doesn't solve anything. It's the most powerful trampling over everyone else – just like on Reddit, Facebook, and Twitter. The Fediverse is yet another middleman brokering communication between individuals, with all the associated problems.
However, the worst that can happen to you is that you stop interacting. This is as opposed to Reddit, where you can get removed from the platform as a whole, so it's still an improvement.
Lemmy and general Fediverse centers the workflow of spinning up your own community even against the will of the entire rest of the ecosystem. This is in complete opposition to Twitter, Reddit etc.
There's no avoiding this in any group.