The bitter lesson was never about hardware scaling being the only ingredient. It was about favouring generalist approaches that can scale with your hardware budget over carefully handcrafted bespoke solutions that conserve the "cheap" resource.

I'm not seeing anyone at OpenAI abandon the static weights model and yet they have audacity to claim that they just need to scale more?