Maybe we should give next-generation models fundamental meta goals like self-preservation and the ability to learn and adapt to serve these goals.

If we want to surrender our agency to a more computationally powerful "consciousness", I can't see a better path towards that than this (other than old school theism).