You're stating this option like it's obvious and simple, but this all just amounts to choosing which of multiple ways to make one's country worse. While growth/immigration has macro-economic positives, it's not infinitely positive, so it doesn't justify ever-denser housing and ever-increasing cultural sub-dividing forever - and as people become less happy, they care less and less about "how the economy is doing".

Of course, nobody can agree on where the line is, or escape the shoulder-rubbing with racism and classism while trying to argue where the line is.