At a certain point, I hope NLP research finally gets that the neurological processes we're attempting to model aren't a single giant unified network either and starts looking at layering models as a way to achieve success and not simply layers within a single model.
Jailbreaking/hallucination issues? Maybe there needs to be the equivalent of additional prefrontal cortex impulse control.
We know from split brain studies that the apparent unity of personality/ego/consciousness is anything but unified.
I have a suspicion we're going to end up getting much more traction in the future with multiple smaller models layered together (more than even GPT-4's rumored MOE) than with a single giant model trying to do everything all at once.
"Moar layers" as the trick to achieve performance may need to be broadened from adding layers of neurons in a single network to adding layers of networks in a mesh, and potentially even layers of meshes.
Jailbreaking/hallucination issues? Maybe there needs to be the equivalent of additional prefrontal cortex impulse control.
We know from split brain studies that the apparent unity of personality/ego/consciousness is anything but unified.
I have a suspicion we're going to end up getting much more traction in the future with multiple smaller models layered together (more than even GPT-4's rumored MOE) than with a single giant model trying to do everything all at once.
"Moar layers" as the trick to achieve performance may need to be broadened from adding layers of neurons in a single network to adding layers of networks in a mesh, and potentially even layers of meshes.