I think it has more to do with the “sea change” it represented at the time. One can easily point to that album as a point in music history from which whole new varieties of pop/rock bloomed. Of course it is never that simple, but history has to be condensed into simple terms to make it comprehensible and consumable.
Regardless of how any single individual might feel about it (my audiophile uncle very much dislikes The Beach Boys), their music can’t be overlooked in the history and a great many people enjoy it to this day — thus standing the test of time to a degree that most music does not.
Yes - genius is a differential, not a scalar. It's not about absolute popularity, it's about cultural leverage.
Beyond his music, Beethoven changed the idea of what a composer did. It was as much a social and cultural shift as a musical change.
The Beach Boys - and especially The Beatles - changed the idea of what pop music could be. There's been an endless stream of imitators copying the tropes that defined the music of The Beatles, but copying a trope makes you an imitator, not an innovator. No one since has had the same cultural impact.
Genius is not a technical exercise. It doesn't even matter if you're unusually competent. (Art and music schools turn out hundreds of unusually competent people every year.)
The real medium is culture itself - concentrating and distilling existing tropes, and inventing completely new tropes that have lasting cultural impact.
Being a novelty isn't enough. Being clever isn't enough. Technical mastery of the medium isn't enough. You only clear the bar by transforming the techniques and systems that other creatives use in a way that matters to audiences for generations afterwards.
The original mix was mono, the stereo stuff is the equivalent of colorizing black and white movies, adding 3-D to a 2-D film or remixing a stereo album into surround sound.
Brian Wilson is apparently deaf in one ear, which would have been less of an issue with mono.
Phil Spector (who was an influence on Pet Sounds) and other producers of the era were aiming to get something that sounded good when played on a car stereo of the time over AM radio and that affected their approach. The Phil Spector boxset was called "Back to Mono".
I think there are professionals who care about recording quality, and then there is an esoteric pop culture holding subjective opinions on sound. Both are labeled audiophiles at times.
Just an aside: Audiophiles are not a homogeneous group because different people value different sonic phenomena. As you likely know, language largely fails to describe such things, and thus there is much debate over the whole endeavor to begin with. Odd that such controversy doesn't arise with sibling human senses such as taste, but humans gonna be human. So it goes.
Regardless of how any single individual might feel about it (my audiophile uncle very much dislikes The Beach Boys), their music can’t be overlooked in the history and a great many people enjoy it to this day — thus standing the test of time to a degree that most music does not.