Not to mention city cleanliness, poverty, the caste system, and plenty of crime...

Caste would not affect a foreigner in India.

In the sense that appartheid wouldn't hurt a white in South Africa back in the day.

That is, if they're insensitive selfish pricks who don't care about the suffering and injustice to others near them.

That's like saying you won't visit the US because of racism. I don't know what you think caste is, but just like racism, it has been illegal for a long time. And just like racism it is still prevalent, but it's probably the worst possible reason to avoid India