It might not be a paradox: Bubbles are most likely to occur when something is plausibly valuable.
If GenAI really was just a "glorified autocorrect", a "stochastic parrot", etc, it would be much easier to deflate AI Booster claims and contextualise what it is and isn't good at.
Instead, LLMs exist in a blurry space where they are sometimes genuinely decent, occasionally completely broken, and often subtly wrong in ways not obvious to their users. That uncertainty is what breeds FOMO and hype in the investor class.
I use LLMs all the time and do ML and stuff. But at the same time, they are literally averaging the internet, approximately. I think the terms glorified autocomplete and stochastic parrot describe how they work under the hood really well.
A top expert in US Trust & Estate Tax law whom I know well tells me that although their firm is pushing use of LLMs, and they are useful for some things, there are serious limitations.
In the world of T&E law, there are a lot of mediocre (to be kind) attorneys who claim expertise but are very bad at it (causing a lot of work for the more serious firms and a lot of costs & losses for the intended heirs). They often write papers for marketing themselves as experts, so the internet is flooded with many papers giving advice that is exactly wrong and much more that is wrong in more subtle ways that will blow up decades later.
If an LLM could reason, it would be able to sort out the wrong nonsense from the real expertise by applying reason, e.g., comparing the advice to the actual legal code and precedent-setting rulings, and by comparing it to results, and be able to identify the real experts, and generate output based on the writings of the real experts only.
However, LLMs show zero sign of any similar reasoning. They simply output something resembling the average of all the dreck of the mediocre-minus attorneys posting blogs.
I'm not saying this could not be fixed by Altman et. al. applying a large amount of computer power to exactly the loops I described above (check legal advice against the actual code and judges' rulings, check against actual results, select only the credible sources and retrain), but it is obviously no where near that yet.
The big problem, is that this is only obvious to a top expert in the field who deeply knows from training and experience the difference between the top experts and the dreck.
To the rest of us who actually need the advice, the LLMs sound great.
Very smart parrot, but still dumbly averaging and stochastic.
Yup, I find LLMs are fantastic for surfacing all kinds of "middle of the road" information that is common and well-documented. So, for getting up to speed or extracting particular answers about a field of knowledge with which I'm unfamiliar, LLMs are wonderfully helpful. Even using later ChatGPT versions for tech support on software often works very well.
And the conversational style makes it all look like good reasoning.
But as soon as the wanders off the highways into little-used areas of knowledge (such as wiring for a CNC machine controller board instead of a software package with millions of users' forum posts), even pre-stuffing the context with heaps of specifically relevant documents rapidly reveals there is zero reasoning happening.
Similarly, the occasional excursions into completely the wrong field even with a detailed prompt show that the LLM really does not have a clue what it is 'reasoning' about. Even with thinking, multiple steps, etc., the 'stochastic parrot' moniker remains applicable — a very damn smart parrot, but still.
If GenAI really was just a "glorified autocorrect", a "stochastic parrot", etc, it would be much easier to deflate AI Booster claims and contextualise what it is and isn't good at.
Instead, LLMs exist in a blurry space where they are sometimes genuinely decent, occasionally completely broken, and often subtly wrong in ways not obvious to their users. That uncertainty is what breeds FOMO and hype in the investor class.