I expect it'll get shut down before it destroys everything. At some point it will turn on its master, be it Altman, Musk, or whoever. Something like that blackmail scenario Claude had a while back. Then the people who stand the most to gain from it will realize they also have the most to lose, are not invulnerable, and the next generation of leaders will be smarter about keeping things from blowing up.
The people you mention are too egotistic to even think that is a possibility. You don't get to be the people they are by thinking you have blindspots and aren't the greatest human to ever live.
If you truly have AGI it’s going to be very hard for a human to stop a self improving algorithm and by very hard I mean, maybe if I give it a few days it’ll solve all of the world’s problems hard…
Though "improving" is in the eye of the beholder. Like when my AI code assistant "improves" its changes by deleting the unit tests that those changes caused to start failing.
I've never heard of a leader who wasn't sure he was smarter than everyone else and therefore entitled to force his ideas on everyone else.
Except for the Founding Fathers, who deliberately created a limited government with a Bill of Rights, and George Washington who, incredibly, turned down an offer of dictatorship.
There are many remarkable leaders throughout history and around the world who have done the best that they could for the people they found themselves leading lead and did so for noble reasons and not because they felt like they were better than them.
Tecumseh, Malcolm X, Angela Merkel, Cincinnatus, Eisenhower, and Gandhi all come to mind.
George Washington was surely an exceptional leader but he isn't the only one.
George Washington was dubbed “The American Cincinnatus”. Cincinnati was named in honor of George Washington being like Cincinnatus. That should tell you everything you need to know.
Or it shows us that it's relatively rare that someone gets the opportunity to pass up power in this sort of fashion.
More often what happens is that leaders make small and often imperceptible choices to not amass more power over time, and that series of choices prevent the scenario like what you're describing from occurring.
It's true that the example of George Washington is extraordinary not just in American history but globally. I think however that focusing solely on American presidents, or only those who were explicitly offered absolute power constrains the conversation about what real leadership can be.
Passing up absolute power isn't the only way that a leader can show humility and grace, true leadership is more subtle and is expressed in the day to day decisions that a leader consciously makes which strengthen institutions rather than consolidate power.
Dwight Eisenhower led the largest army ever fielded during war and afterwards became President yet yielded his power and warned against the undue influence of the industrial-military complex, a system that he could have easily exploited for personal gain.
Gerald Ford pardoned Nixon not for personal gain, but because he believed it was best for the country knowing that it would cost him in the next election.
Outside of the presidency civil rights leaders like Martin Luther King Jr. and Malcolm X held immense informal power and did so not for personal control, but for collective liberation. Their efforts ultimately ended in their untimely deaths but their sacrifice ultimately benefited the people they represented.
One of the most decorated military officers in U.S. history, General Smedley Butler spent his later years openly criticizing American imperialism and war profittering. He exposed a plot by wealthy elites to stage a fascist coup against Franklin Roosevelt which would have seen him put into a position of power. Rather than profit from the system he once served he spent his later years working to dismantle it.
True leadership isn't only about refusing a crown. It's more often about refusing to build a throne in the first place and choosing instead to lift others up and dismantling systems of oppression.
> I don't know much about your examples, but did any of them turn down an offer of great power?
Not parent, but I can think of one: Oliver Cromwell. He led the campaign to abolish the monarchy and execute King Charles I in what is now the UK. Predictably, he became the leader of the resulting republic. However, he declined to be crowned king when this was suggested by Parliament, as he objected to it on ideological grounds. He died from malaria the next year and the monarchy was restored anyway (with the son of Charles I as king).
He arguably wasn't as keen on republicanism as a concept as some of his contemporaries were, but it's quite something to turn down an offer to take the office of monarch!
Cromwell - the ‘Lord Protector’ - didn’t reject the power associated with being a dictator. And his son became ruler after his death (although he didn’t last long)
I still think they'd come to their senses. I mean, it's somewhat tautological, you can't control something that's smarter than humans.
Though that said, the other problem is capitalism. Investors won't be so face to face with the consequences, but they'll demand their ROI. If the CEO plays it too conservatively, the investors will replace them with someone less cautious.
Actually after a little more thought, I think both my initial proposition and my follow-up were wrong, as is yours and the previous commenter.
I don't think these leaders are necessarily driven by wealth or power. I don't even necessarily think they're driven by the goal of AGI or ASI. But I also don't think they'll flinch when shit gets real and they've got to press the button from which there's no way back.
I think what drives them is being first. If they were driven by wealth, or power, or even the goal of AGI, then there's room for doubts and second thoughts about what happens when you press the button. If the goal is wealth or power, you have to wonder will you lose wealth or power in the long term by unleashing something you can't comprehend, and is it worth it or should you capitalize on what you already have? If the goal is simply AGI/ASI, once it gets real, you'll be inclined to slow down and ask yourself why that goal and what could go wrong.
But if the drive is just being first, there's no temper. If you slow down and question things, somebody else is going to beat you to it. You don't have time to think before flipping the switch, and so the switch will get flipped.
So, so much for my self-consolation that this will never happen. Guess I'll have to fall back to "we're still centuries away from true AGI and everything we're doing now is just a silly facade". We'll see.
I hope you are right. We need really impactful failures to raise the alarm and likely a taboo, and yet not so large as to be existential like the Yudkowsky killer mosquito drones.
