I think you are being unreasonable, misunderstand the problem space and wildly overestimate what a computer system is capable of doing. You cannot write a Python program to fix global warming or replace corrupt government officials.
>You cannot write a Python program to fix global warming or replace corrupt government officials.
How can you say that you're appreciating the scope and seriousness of superintelligent AI as a concept when you are comparing it to a Python program (a pithy shorthand for a small-in-scope, trivial computer program)? Saying "it's just a computer system" feels like a category error when everyone who talks about superintelligent AI's impacts talks about it with respect to its integration into real-world systems, embodied robotics, acceleration of manufacturing, mass manipulation, etc. Is the bias just based in "computers don't affect the world that much so anything under the category of 'computer' can't affect the world that much"?
Your first comment in this chain was that there was no point in considering what to do now because ASI would be able to outsmart humans in any domain, so how come now you're saying that ASI couldn't do anything substantial in the real world? Everything humans have ever done substantial in the real world has been as a result of our intelligence, coordination, and ability to create and use tools, why wouldn't superintelligent systems, bolstered with the same aptitudes, be able to do the same?
1. You and I both clearly agree that superintelligent AI doesn't exist. Anyone that's "seriously" talking about superintelligent AI has not seen it and is extrapolating from guesswork. Their credibility on this subject directly correlates to how seriously they understand AI from a technical perspective. Right now, we are eschewing every technical detail to assume that somehow, superintelligence exists one day. I cannot give you any further benefit of the doubt.
2. AI requires heuristic variability. It cannot operate or generate unique responses without seeding some form of entropy. Regardless of whether you think AI is smart or dumb, this creates a system of mutual distrust where researchers will never know whether they've created superintelligence. ChatGPT could give you the solution to world peace right now, and you'd probably re-roll it because it's not exciting enough. Conversely, many people will take superintelligent responses too seriously and beg for fake solutions to issues that cannot be resolved. We cannot square the issue of "consistently correct" and "always different" with AI.
3. We already have a framework for evaluating AI impact, it's called "the last 5 years of your waking life". You may notice that AI isn't replacing much of anything. Your favorite YouTubers and musicians are still human. Your favorite joke is still decades old. If a superintelligent ChatGPT materialized overnight, I don't think any of that would change. Maybe that's because we're dumber than the AI, but maybe it's also because we have irreconcilable differences.
