1. You and I both clearly agree that superintelligent AI doesn't exist. Anyone that's "seriously" talking about superintelligent AI has not seen it and is extrapolating from guesswork. Their credibility on this subject directly correlates to how seriously they understand AI from a technical perspective. Right now, we are eschewing every technical detail to assume that somehow, superintelligence exists one day. I cannot give you any further benefit of the doubt.
2. AI requires heuristic variability. It cannot operate or generate unique responses without seeding some form of entropy. Regardless of whether you think AI is smart or dumb, this creates a system of mutual distrust where researchers will never know whether they've created superintelligence. ChatGPT could give you the solution to world peace right now, and you'd probably re-roll it because it's not exciting enough. Conversely, many people will take superintelligent responses too seriously and beg for fake solutions to issues that cannot be resolved. We cannot square the issue of "consistently correct" and "always different" with AI.
3. We already have a framework for evaluating AI impact, it's called "the last 5 years of your waking life". You may notice that AI isn't replacing much of anything. Your favorite YouTubers and musicians are still human. Your favorite joke is still decades old. If a superintelligent ChatGPT materialized overnight, I don't think any of that would change. Maybe that's because we're dumber than the AI, but maybe it's also because we have irreconcilable differences.
2. AI requires heuristic variability. It cannot operate or generate unique responses without seeding some form of entropy. Regardless of whether you think AI is smart or dumb, this creates a system of mutual distrust where researchers will never know whether they've created superintelligence. ChatGPT could give you the solution to world peace right now, and you'd probably re-roll it because it's not exciting enough. Conversely, many people will take superintelligent responses too seriously and beg for fake solutions to issues that cannot be resolved. We cannot square the issue of "consistently correct" and "always different" with AI.
3. We already have a framework for evaluating AI impact, it's called "the last 5 years of your waking life". You may notice that AI isn't replacing much of anything. Your favorite YouTubers and musicians are still human. Your favorite joke is still decades old. If a superintelligent ChatGPT materialized overnight, I don't think any of that would change. Maybe that's because we're dumber than the AI, but maybe it's also because we have irreconcilable differences.