Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I kept that drawing up for many months after, and would gaze at it often during meetings and idle moments in wonder.
This is to say: the autoregressive decoder-only transformer llm architecture as pioneered by openai is wildly simple for how revolutionary its results are. I was reading about non-learned classical SLAM systems (uses video + handcrafted math to produce 3d mappings of physical spaces while also locating the camera in those spaces) at the time, and comparatively speaking I’d say the math is about as complicated as ONE of the components in those complex formulations. The only reason frontier LLMs need 6-figure computers to run is because the model designers made the middle bit in those models REALLY BIG, dimensionally speaking. They just took the steam engine, made a few gargantuan versions of it, and are selling them as the ultimate source of power.
This was openai’s entire breakthrough. Making this particular model architecture larger leads to emergent capabilities like being able to pick the best ending to a story/set of instructions or answer questions about broad factual knowledge. I’ve been meanwhile watching these AI companies attempt, successfully, to sell this capability as some sort of robot consciousness hand-crafted by supergeniuses. The fact that they are getting away with it is almost as shocking to me as the discovery itself.
This interview https://youtu.be/oWOz2htozfI?si=qdQ0uZRoZOYeThOn
from 2 days ago with a top researcher from OpenAI directly addresses the bitter lesson argument and the importance of scaling for the history of their models.
So the take-away here is that we (as humans) try to model these AIs like humans, but eventually these AIs get better. Which to me seems like a logical conclusion if they can do "things" (like "learning" or pattern matching) much faster than we can (the compute).
Then language in LLMs is a bottleneck, the AI is constrained by the language, and thus if we want to scale further we could let AI create its own language (we would then have to translate whatever it creates back to a language we understand).
It is the same for instance if we check the language of Inuit (people who live in the north and make temorary shelters like igloos in the snow) they have multiple words/verbs to describe the snow, while in English we only have one (?): snow. In English we don't need more words (we can explain snow state using multiple words) but for the Inuit language it makes sense to create these new terms (would also make it easier and faster to communicate).
So in some sense, all languages are then "newspeak" to whatever a general language is what researches or AI might come up with.
If this sounds dumb let me know, but if you know some research in this general language direction (I'd assume general AI research) would love to see it!
I think one hypothesis along these lines is that, if allowed, due to the limitations of human language you described, LLMs will gravitate towards "inventing" their own language (which, due to training pressures, may even resemble english from the outside, but contain deeper, "true", meaning within), but that we should do our best to prevent this even if it bottlenecks reasoning capabilities since it would cut off our ability to read its "true" thoughts and detect misalignment
Chain-of-thought (CoT) reasoning models “think” in natural language understandable by humans. Monitoring their “thinking” has allowed us to detect misbehavior such as subverting tests in coding tasks, deceiving users, or giving up when a problem is too hard.
We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future.
We have further found that directly optimizing the CoT to adhere to specific criteria (e.g. to not think about reward hacking) may boost performance in the short run; however, it does not eliminate all misbehavior and can cause a model to hide its intent. We hope future research will find ways to directly optimize CoTs without this drawback, but until then
We recommend against applying strong optimization pressure directly to the CoTs of frontier reasoning models, leaving CoTs unrestricted for monitoring.
We understand that leaving CoTs unrestricted may make them unfit to be shown to end-users, as they might violate some misuse policies. Still, if one wanted to show policy-compliant CoTs directly to users while avoiding putting strong supervision on them, one could use a separate model, such as a CoT summarizer or sanitizer, to accomplish that.
My understanding is that the Inuit snow claim is a bit of a myth. Beyond English words like slush, sleet, powder, hardpack, flurry, blizzard, etc you can also say "fluffy snow", "wet snow", etc. The Inuit language is just smooshing the adjective so you get something like "wetsnow" as one word.
not exactly, bitter lesson is one meta-level up from "scale eats everything". this is a common misunderstanding of bitter lesson that rich sutton has been fighting ever since the thing was written. in rich's own words[1], the modern summary is
> Don’t be distracted by human knowledge, as AI has been historically.
> Instead focus on methods for creating knowledge that scale with computation, like search and learning.
so the lesson is choose methods that scale with computation, not just that blindly scaling up anything (data, params, people, whatever) works, it is choosing the right x axis and the right scaling laws consistently wins out in the long run despite short term wins from other methods.
Indeed. It's pretty interesting to realize after implementing GPT-2 that the frontier models are scaled up versions of that, with various tweaks to improve performance, model-wise.
The secret sauce though is all the datasets, RL training, knowledge of what works from doing all kinds of ablation experiments, and a massive compute moat.
The secret sauce is also having the necessary 'creativity' to not get ceased and desisted into oblivion and jail from all the copyrighted material you trained your model on. Btw, not making a moral judgement, [0] shows Michael and Dalton from YC discussing why Ilya Sutskever had to leave Google to pursue what's now ChatGPT
There is a whole moral judgement to be made here...lets hope Ilya wont get too pissed off if somebody leaks the work of his new initiative...information wants to be free and all that...
Also would love to know if the same Legal team advised on Gemini...
How do we know that today's frontier models are merely scaled up versions of that? Genuine question, since the labs have narrowed what they share over the years to now almost nothing, in terms of how the model was trained and how it works under the hood.
We know for sure the architecture of the open weights models since llama.cpp understands the architecture it needs to build to plug the weights into to run them. It's always possible that the latest closed model is doing something architecturally different than the open weights ones we know about, but judging by how close the large open weight models such as DeepSeek are to SOTA performance, this seems unlikely. When OpenAI first came out with their near-mythical "Strawberry" (aka "o1") thinking model there was all sorts of speculation that they had made some sort of architectural breakthough, but then DeepSeek replicated the capability and published how they did it, proving that it was just better training, not any architectural change.
There have been minor changes to the architecture over the years, but these are basically all efficiency tweaks such as various types of attention (some pioneered in the open by DeepSeek) that better scale to large context lengths, and the confusingly named "mixture of experts" architecture, but what's more notable really is how little the architecture has changed. The capability gains have been coming from better training and better data.
- R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models)
Then, there's the incentive analysis. If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare. I'm not saying there's noone cooking a new architecture, just that it is a pretty rare event. And it would have to come from some researchers that would be happy to not publish their findings, which is not really what a sizable portion of elite researchers (obviously not all) are incentivized to do.
Of course, it's a bit of a verbal compression to claim simply 'scaled up'. They are recognisable scaled up transformers, but most new models come with a few tricks, but we're at the point where those usually are not an architectural rewrite and added to solve an explicit problem, like hallucination, not for big new capability gains.
No they are clearly not just scaled up versions of gpt 2; there are different LLM architectures like mixture of experts etc that appeared relatively recently. I am not an expert though, far from it.
separately trained experts can surpass performance in their activated regime and DOES result in a smarter model, the Claude system cards talk about this and eg there is https://openreview.net/forum?id=iydmH9boLb to read...
If you can make the existing model faster, you can then save your inference budget to then make your model bigger, which then makes it smarter.
A lot of how smart the models can be comes down to budget. If you can make your existing thing cheaper, you can instead make it bigger for the same price.
Not really “smarter” though? It’s just a big probability engine.
(Not trying to flame bait or anything. I just wouldn’t call LLM as exhibiting intelligence. It is great at making connections based on probability but doesn’t have a semantic understanding of what it is doing)
> The secret sauce though is all the datasets, RL training, knowledge of what works from doing all kinds of ablation experiments, and a massive compute moat.
ReAct loops and tool-calling are the critical development feature. They turn a model from something that generates text into something that can independently influence the world around them.
The big breakthrough is we can interact with the agents using natural language - because of the LLM.
It is the combination of LLM and agent-harnesses that make it look really smart. Agent-harness is a programmatic device that lets us tap into the vast knowledge in the LLM.
It is probabaly true that many TV-commentators fail to appreciate this fact and therefore think LLMs are super-intelligent. No, it is the combination of LLM and the programmatic agent-haness that is the breakthrough.
An interesting thought is that the LLM could in theory code the agent-harrness, start it running every time we interact with it. Currently the agent-harrness I think is pretty static I think. In theory it could be dynamically created for every task. Would that make it better don't know.
> The big breakthrough is we can interact with the agents using natural language - because of the LLM.
Without ReAct and tool calling, all you have is a chatbot. That's useful, but it's just a chatbot.
ReAct loops and tool calling is what unblocks high value usecases. It enables systems to actually address free-form problem statements, gather data that is not a part of their training set, inspect the current state of services,and trigger actions in external systems. This goes well beyond mere chatbots.
> It is the combination of LLM and agent-harnesses that make it look really smart.
It's really not about "smart". It's about autonomous systems, and being able to consume and analyze new data, and trigger actions in external systems.
There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and the math tricks like the residuals and so forth. But the bottom line of all this is that the Transformer works because of the incredible semantical power of simple/huge FFNs.
You can go another step - a FFN can be simulated on a Turing machine, thus it just exemplifies the incredible semantical power of the Turing machine model of computation. (in fact you don't even need a Turing machine, since there is no looping in one forward pass).
In theory you can run a huge FFN on the tiniest Turing machine, in practice it's much better to run a Transformer on the latest NVIDIA hardware. Or as they say "quantity (performance) has a quality all its own"
I was about to post your last point / quote. Going multigpu is relatively not so though but once you go multi-node you have distributed storage/io/compute system which is highly non trivial. Add that the long training times now you have robustness/fault-tolerantness concerns with hardware failures and restarts. Today’s training systems are engineering marvels.
Yep. It's nearly identical to the neural nets we were using in the 90s. Back then even a supercomputer wasn't big enough or fast enough to do what we do today.
I have to wonder though. Is this all a human brain is? A similar thing to an LLM just scaled exponentially larger. I mean a brain is not just neurons with simple connections to each other. The neurons, axons, dendrites, <insert_unexplained_thing>, etc in a brain are all holding and processing information in different ways and doing it nearly 100% in parallel. That's a really big model.
The biological discoveries show how complex a biological brain actually is. Even the tiny brains in a bee or spider are able to solve puzzles and use tools. That's crazy.
No, it’s definitely not what a human brain is. That makes very little sense. The ways we interact with language (and thus conceptual memory) is completely and fundamentally different.
Chomsky did the opposite of what you're saying. He didn't ignore spoken language. He said that human vocalization is independent of language, and that the way our brains can manipulate and use sound (a cognitive capability, not specifically an aural one) is the fundamental differentiator that allows us to make compound ideas, and our specific use of language is a byproduct.
Example: a programming language's capability to produce complex software does not come from some inherent quality of language. It comes from binary. 0's and 1's, representing basic logic, and that being built on top of with an abstract "tool" called a language. If the binary logic didn't work, the language wouldn't do anything.
A dolphin can make sounds, and technically has a language, but they can't manipulate or recursively compound concepts (as far as we can tell) in order to create modified ideas. If they could, they probably would have come up with vastly more advanced fishing methods than the (admittedly novel) ones they have now.
It is different, but there may be some universal principles that are relevant more abstractly among both cases. Of particular interest is the empirical notion that statistical models of a certain form will always tend to "average out noise" and "learn meaningful patterns" up to the capacity that those models have for representing said patterns. A parallel notion to this is the hypothesis dubbed "thermodynamic origins of life". The universal principle binding these two seemingly disparate topics is one that seems to underlie any sense of "learning" in physical systems: that semantics of those systems depend on their representational power, and the semantics they do come to represent are the results of adding up many pushes in one "direction" (phase space / state space / etc.) encoding a pattern, and adding up many random noise jiggles will cancel out but give you a first-order sense of variance of those semantic features as expressed by the environment.
As this description is so overly abstract, an exercise for the reader is to try to work through an explanation of how, say, a river delta comes to "learn" about its environment by "reacting" to the influences at its borders, and how it "encodes" whatever it is that it learns in the substrate that it inhabits.
I suspect it was considered many times, but the sheer computation scale would make it feel like obscene brute force. It feels like the right shape but too wild to think about implementing.
I think as time went on, and hardware got better, it seemed more reasonable to actually think about a viable implementation of what I think was a widespread intuition anyone in ML had that everything's context is everything.
It just seemed like a theoretical thing until hardware caught up. Maybe. Perhaps I'm applying a retrospective excuse to why it took so long.
People definitely wanted to train deep networks before, but didn't know how. They evdn tried things like training layers independently.
I don't think it was intuitive to anyone back then, the vanishing gradient problem was a big deal since the dawn of NNs. I'm not sure what you mean by sheer computation, residuals allow you to have deep networks instead of shallow and wide ones. You can have equivalent parameter count.
What hopes/paths does a mere CS bachelor (not deep into stats/maths), and mid level dev (native mobile only; 10-15 years exp.), have about not only understanding it (maybe not fully) but getting possibly into this as a career? Not expecting churning out models and AI systems from the first weeks/months but entry/employment into this field?
(If I can be honest, and I am not being disparaging about anything lest it might seem so, I am looking at it from a career breakthrough/move perspective rather than an intellectual pursuit.)
> I think you need to ask what you actually want to do with the AI.
What about improving the efficiency of token consumption, etc., basically opportunities for improving cost/performance?
I keep thinking there has to be a better way to share context with models than dumping entire gigantic skill files of raw text or otherwise into them - I'm betting there's a bunch of low-hanging fruit there.
There may be some low hanging fruit, but they're not available to people without deep understanding of how the math works. Well paid people already spend a lot of time thinking about this.
i am not sure acctually of the math is acctually that complicated/important. the math around neural networks is calculus/chain rule etc and for model comparison/validation one needs statistics. the required math for e.g. understand transformers is quite accessible.
Im also a mere mortal, and after putting a few years into it IMO I’d say people make it much more complicated than it actually is. I failed most of my math courses for lack of interest, but found passion later with the aforementioned SLAM stuff. I have no doubt you or any other programmer could learn this stuff, especially since you can ask ChatGPT clarifying questions.
I have no idea about careers at this point, I’m still doing fancy IT work as my day job I and look away from the future with dread. I also haven’t been looking for new roles on the open job market, so who knows maybe there’s multimillion pay packages for anyone who can articulate how attention works in an interview.
I have a BS in CS (and have been in the field for 25 years). I couldn't understand the transformer architecture until I built a few myself. Here are the books I worked through. I now feel I have a very good understanding of modern LLMs.
I went this route because I had difficulty visualizing the content of the Attention Is All You Need paper. After going through both books, I can now understand every part of that paper.
I'm currently working on a robotics project that uses Nvidia's GR00T N1 model, and I was able to understand the research paper. [0]
"Attention is all you need" is actually a bad paper if you want to learn about autoregressive LLMs specifically, because it describes a more complicated encoder-decoder architecture while modern LLMs are decoder only. So it's an unnecessarily hard way to get into the subject. "Language Models are Unsupervised Multitask Learners" is probably what you are looking for (aka the GPT-2 paper). This was the first time LLMs really showed what is possible, i.e. they can learn to generalize very well from unstructured data. So no more human labelling necessary, which until then was the primary bottleneck in ML. The paper also lists several key ingredients beyond transformers that are mostly still in place today. This also highlights that there was more to it than just "scaling the transformer algorithm" like many people claim. Most developments since then were about improving training data, until "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" drastically changed the architecture landscape again. Later big developments like thinking/reasoning/chain of thought/inference time compute (whatever you want to call it nowadays) are actually all about training again. They work using the exact same architecture.
Chain of Thought was kind of an obvious solution that everybody knew was necessary by the time chatgpt / gpt4 came out. It was just a matter of time that frontier labs actually shipped it.
MoE was also pretty straightforward, just a bit surprising how well it worked (that you can get away with just 1/32 active parameters), but most researchers would have come up with it on their own probably.
The true ground breaking papers are the first two you mentioned (transformers and gpt2), and InstructGPT was also very surprising that it worked so well.
Reasoning is a little bit more than just "baked in" chain of thought prompting. The important takeaway here was that it is not realized at the architecture level of the neural network. And you could say that all these things regarding LLMs were pretty straightforward. But only in hindsight, otherwise there wouldn't have been so much time and effort spent on intermediaries. Breakthroughs mean people simply didn't know stuff before, even if it seems easy with the benefit of hindsight.
I had the same reaction as you, when I learned in detail, how all this works. But then I also learned about superposition and compressed sensing, and now...I am not so sure anymore...
It's not just the architecture but also the data - the decoder only approach lets you train in parallel over blocks of text (no RNN serial waiting), that allows you train on much, much more data.
For anyone who is curious about the first paragraph here, this is actually a great video overview of how LLM works and the tokenization part.
Tangentially related: This part always seemed fuzzy to me, especially when dealing with data scientists and how they talk about how 'ML' looks at problems. I had this issue when working at a SIEM vendor where they kept going on about use case development having to be designed a certain way to catch things. It was all very frustrating.
I’d recommend my method of just drawing out the block diagram and drawing out + digging into the math at each step! I’m the kind of person who needs to take time to ask lots of questions before stuff clicks, and if you are too I strongly recommend it.
I picked it up from trying to teach myself that SLAM stuff. The papers are very short, but highly information dense and at the time there was no ChatGPT to help me. I got through them by just creeping my way through the math with a whiteboard, and something about drawing it out and having it there in my office made it all click. Trying to watch piecemeal lectures on YouTube or grind through foundational books like MVG just didn’t work for me, I used them instead as references for my drawings.
Same happened when I tried learning this GPT stuff. karpathy’s videos were out at the time, but I couldn’t really stay focused on them or connect the math with the code. Most other descriptions I could find were focused on getting you to use their inference library or harness. Assembling the picture together on my whiteboard by focusing on drawing out the block diagram continues to be my personal favorite method for deep understanding of complex systems.
Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books.
I don't think there is anything in a transformer I couldn't explain in the smallest detail now.
Well, as I suggested, working through the implementation yourself will give you that intuition. That said, I think the simplest way to explain why positional encodings are useful is that it gives the transformer just enough information to make attention meaningful without negatively impacting any parallel, content-based comparisons.
A vanilla self-attention layer is just a set of token vectors. Without positional info, swapping two identical embeddings changes very little about what attention can compute. We can "fix" this problem by using positional encodings. Text that has meaning isn't just a set of characters; the location and order of those characters is what provides meaning.
I mean there is a little something called compute. And other complexity that comes like writing code to efficiently distribute a model across machines.
> Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I kept that drawing up for many months after, and would gaze at it often during meetings and idle moments in wonder.
how did you know about the steps and there was math involved. i am curious about your process and you came up with what exactly to learn to unravel the mystery.
Architectures have evolved significantly since then. DeepSeek v4 =/= GPT-3. Even then, a great deal of complexity lies in everything surrounding the architectures e.g. how do you implement them performantly on modern accelerators, how do you distribute the model across a set of accelerators, how do you post-train, etc. And pre-training itself is a dark art. If you legitimately think that frontier labs are doing something equivalent to whatever you wrote on your whiteboard, you’re clueless.
We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is also rather mysterious. My eldest son's first word was to actually imitate a cow mooing, and then after that to imitate a motor noise of a tractor or truck. And then after that a meow. (His first complete sentence was "King Graham fell"...)
My next child took a completely different path to language, including skipping all the non-verbal imitations.
And then at some point, you just suddenly can two-way communicate with them when you couldn't before, and then after that, they can engage in reasoning.
It’s interesting to me how similar attempting to understand LLMs is to neuroscience.
“When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome”
We’re basically just probing around and trying to reverse engineer an emergent system.
To your point, this system may be quite different from model to model (human to human) although some similarities likely occur.
The comment I was responding to tried to belittle the OP’s understanding of transformers, by mentioning that running an LLM at scale is much harder than the simple white board diagram.
My point was simply that we don’t know why they work, and all the extra optimizations isn’t the “thing” that makes it emergent.
Simply scaling the “GPT” is good enough to see it, so the OP’s awe should stand.
(On a side note, what other architectures can we scale to find similar emergent behavior?)
Isn't the LLM simply predicting what should be the next sentences after user's input, using its algorithm and data it has exatrcted from existing texts on the internet. The algorithm that does that could have many different designs, some better some worse for the purpose of predicting what output makes most sense next?
So what is it that we don't understand about why theyr work? The algorithm? We have the code. Why the specific algorithm makes such good predictions? I see it as a generalization of trying to predict who wins Kentucky Derby.
Human brain capabilities are truly amazing, imagine if people didn’t treat their children as if they are stupid and didn’t constantly lie to them, because kids are stupid right, they wouldn’t understand. What heights could be reached.
We don’t treat children like they’re stupid, we treat children like they’re children. A stupid adult is treated very differently than any child.
Adults are expected to have their world models approximately correct in terms of physical environment so they won’t accidentally kill themselves by falling off a cliff; then there are the social norms which adults are expected to conform to so everyone is kinda predictable to everyone else so adults don’t kill each other too often over food or mates. Understanding of neither is expected from children.
Another example, my parents taught me to read at about 4 years old. When I started kindergarten (the year before 1st grade in the US), the teachers and principal didn't believe I could read and I had to prove it by reading a book to them I'd never seen before.
I think they're right that kids (at least in the US) are generally treated as less capable than they are, and it ends up slightly delaying their development.
You may have been raised properly since you don’t get what I mean. I really envy kids with “Chinese parents” that had them learn math early on and not some bullshit like that if you put your tooth under your pillow, then a tooth fairy will come.
Maybe math works but critical thinking doesn’t. There are people who have lived for many decades without ever questioning insane b.s. they were taught as kids.
They aren't stupid, but they aren't quite ready to handle the full responsibilities of the world and worry about things they don't need to worry about.
My son is very worried about black holes lately when he learned anything that goes into one can't get out. He's pretty concerned astronauts could get stuck in one some day. So I explained to him that Hawking radiation does actually mean you can eventually get out; it just takes some time.
I didn't think it pertinent to mention spaghettification, the fact anywhere near a black hole will be really hot, or that cosmic censorship means whatever Hawking-radiates from a black hole wouldn't be an astronaut anymore.
It was also fun to hear Hawking speak. He wanted to know if Hawking was a robot. I said no, but he has a robot talk for him. Not quite true, but close enough.
Waste 22 years of life without learning anything and then slave away at a 9-5 job you hate. Brilliant strategy. At least you had “fun”. Then blame billionaires or something.
Childhood only lasts 13 to 15 years where I am. By the time you’re in high school, you can be expected to be responsible in some matters. By 22 you have 7 years of experience in making decisions for yourself.
Statistically most likely in what context, given which preconditions? Because each prompt sequence is unique so the probability of any token following it is unknown.
Correctness is based on axioms and rules. You need to define your axioms and rules first before you can determine correctness.
If you’re talking about matrix multiplication, I can use mathematical rules and axioms and proves formally that the multiplication is correct. For next token prediction, I can prove that the set of tokens is finite and that the next token is always part of that set.
But things like grammar correctness, or semantic consistency over a few sentences are not hardcoded rules in the model. They’re emergent properties, mostly due to the amount and quality of data available for training. Quantization is mostly about how much we can shed without loosing a particular emergent properties (like dithering or psycho acoustic audio compression)
This "they just predict the next statistically most likely token" is such an handwavey and willfully misleading explanation, it's unreal, and I'm so fucking tired of seeing it so incessantly repeated. It's beyond asinine.
You know it perfectly damn well that a typical person's idea of statistics is not some insanely high cardinality stateful prediction, but a "well a coin toss is a 50:50, and a lottery win is a 1:100000000". You also know it perfectly damn well that as a result, people will just think that all the sentences chatbots ever produced to them were then just somewhere in the massive training set, letter by letter. This insinuation is often even explicitly appealed to.
And that picture is outright false. It's a statistical process, yes, so saying that it does what it does by "just doing statistics" is gonna be a generally correct description, but that's not at all inquisitive to how exactly does it do it, nor is it the zinger you think it is. If you did the aforementioned, you'd just get milquetoast nonsense, like you can see in the countless Markov-chain primers. And while the models do have a lot of the training set lossily captured, they do also absolutely generalize (that's how they can do that lossy compression), and you can quite literally find representations of those generalizations in them, and also see them activate.
It's like summarizing how any program works by just saying "well it just manipulates ones and zeroes". Not very informative, is it? Or how programs are written by just programmers sitting in a cushy office, ryhtmically pressing keys on a keyboard. Not a very fair or insightful description, which you'll know if you've done any amount of programming in your life on your own. Extends to all other white collar jobs too.
It's also not even true in the most literal sense: models can and do absolutely choose a less than maximally likely next token, that's what the various decoding parameters are for. "Maximally likely next token" further conviently skipping over how that likelihood is established in the first place, i.e. the literal point of the question, going in a cute little circle.
LLMs predict next token one at a time. (Stochastically.) Literally. It's what they do. That's how they literally work.
If you don't believe me, download llama.cpp and see for yourself.
P.S. I write inference backends in C++ every day. The gall of people like you who figured out how to prompt Claude and think they're hot shit now is simply unbelievable.
So you work on inference engines, and don't see at all what'd be hilariously disingenuous and reductive about describing how LLMs operate as "just parroting the most statistically likely next token"? It is literally* what they do, yes. And only literally, with a big asterisk of "non-colloquial meaning" after the word "statistically". Like how "significant" means something pretty different, albeit related, in academic writing vs everyday speech.
It's equivalent to professing how you just make apple pies from scratch, while your first step is to always reinvent the universe.
You're further magically blind to this operational fact being weaponized as a trope for furthering anti-ai sentiment (i.e. that it's a political dogwhistle at this point), and to thus you participating in that every time you repeat it?
* Ignoring the decoding caveat I already mentioned, along with the countless ways they're steered. There isn't jack that's likely about some of the responses they produce, and intentionally so. Including the whole chat partner act.
It's so beyond tiresome. It's a classic case of someone being technically correct, and abusing the gap between that, and what people actually gather from it, for sentiment manipulation (willfully or otherwise). And I have a pretty hard time believing at this point that it's the otherwise.
I really don't know what's so interesting about auto-complete or next token prediction that it captures these people's attention so much. They're so blatantly not the salient quality to these products that is of interest to the common discourse, it's just baffling.
I help write optimized CUDA kernels for proprietary hardware. They may "literally" work this way, but that is quite besides the point.
If you don't see why then you have exactly demonstrated my point in how practitioners like you simply lack the foundational understanding in philosophy, information theory, human consciousness, human cognition, neuroscience, necessary to bridge this conceptual gap.
(Rather, it is that we know so little of how consciousness or what intelligence even is, that we cannot possibly use first principles to preclude LLMs from possessing these qualities)
You don't understand the argument, so you keep repeating first order mechanistic observations that are irrelevant. If you don't want to understand the argument, don't be surprised when people refuse to engage with you, especially when it's evident to those more knowledgeable the position you hold is the ignorant one.
Great, and won’t we all be just as surprised when human self-attentional control turns out to be just as simple or just as complex! Our minds as a strange fabric built of threads of recursions without the benefit of any explicit clock.
How do you verify someone’s age reliably without identifying them? Unless there’s some standard around zero-knowledge proofs they’re implementing that I’m not aware of, they’re probably going to end up identifying everyone as part of this system, since kids will try and bypass it and parents will demand it be made more robust. Kids will still bypass it no matter what we do.
You don't. The law we're discussing doesn't verify someone's age. Please stop calling it an age verification law, because that's completely disingenuous FUD.
I don’t understand how video evidence from a mass surveillance network would have helped here. They found your car without it! Shouldn’t your issue be with the prosecutor, and thus your ineffective local government?
Otherwise, what’s to stop them from just telling you video evidence isn’t enough, because jurors have become accustom to thinking that video evidence can be faked by vindictive cops?
Really hate to say it, but I’ve stopped publishing my work too for this reason. I spend most of my time now building my own little software ark, and I aspire to no longer think of programming in the next few years. I feel like the creative economy in general will be unrecognizable in the near future, maybe nonexistent. I wonder what modes of collaboration on ideas might form in the next few years.
Here is what the purveyors of AI don't seem to realise. You can bend copyright law all you want in order to train your models on whatever you can grab, but in the absence of genuine protection of their creative work authors are simply not going to be publishing at all.
I think they see it all too well. They still think they can make bank today while it lasts, whatever comes after is some other shareholder's problem. And if we're talking about open source, killing it might be a positive side effect, they'll be ready to sell you a closed source alternative when you no longer have options.
Furthermore, if people not only stop publishing, but also take down already published works, it will create a moat around already existing Language Models
And the more they DDOS small websites — instead of respectfully scraping once — the more realistic my conspiracy theory looks.
Without any material or immaterial benefits? And with one's work being ground up and turned into weights for the next version of the machine that's threatening one's employment?
> People who are making stuff because they want to share it are still going to be publishing.
Those people who do that are too few and far between to make a difference. The majority of open source devs aren't giving away the source without a license. That license is how they specify what they want in return.
> The majority of open source devs aren't giving away the source without a license.
100% of open source devs aren’t giving away the source without a license, since a licence—the grant of permissions for what is otherwise exclusive to author under the law—is what makes something open source.
> That license is how they specify what they want in return.
No, the license is how they legally give away permission to use material that is legally subjejct to their exclusive rights by virtue of creation. The license may be a contract license that, as you suggest, involves mutual exchange of value, but for many (especially permissive) open source licenses it is a gratuitous bounded grant of permission which has limits but does not involve giving something of value back to the creator.
> No, the license is how they legally give away permission to use material that is legally subjejct to their exclusive rights by virtue of creation. The license may be a contract license that, as you suggest, involves mutual exchange of value, but for many (especially permissive) open source licenses it is a gratuitous bounded grant of permission which has limits but does not involve giving something of value back to the creator.
Wrong. What they want in return is either credit or derivatives of the software. It's disingenuous to suggest that all these authors specifying, in a legal document, the exact mechanism by which to pay them back don't know what they are asking.
If you're not happy with that trade, then don't make it.
The sad thing is I feel trapped on all sides of the debate, I wrote a book about LLMs and human creativity (spoiler Humans win for a long time) but I was going to do it as a blog series, instead I published https://www.amazon.com/dp/B0GXCSY4W8 because I felt at least I might get a bit back for literally 100’s of hours of my life I poured into the book and my editor and friends who read and provided reviews.
And I push a lot of open source code including a ton for the SWGEmu project, but now I’m of mixed mind to stop pushing anything public. I can’t decide, am I talking out of both sides of my mouth, it’s a confusing time to navigate for sure.
Indeed sad, congrats on publishing your book though. I’ve certainly felt a bit of that same angst myself.
I think SWGEmu (cool project, just learned of it from you!) do represent some optimism though. Maybe these sorts of passion projects will take over the space?
I don’t think people realize that these devices can even be used that way. I talk with people outside of the tech scene frequently, and they are routinely surprised when I tell them about this sort of capability. The ring doorbell Super Bowl commercial about finding lost dogs was a genuine shock to people! I think there’s a degree of visibility you need to get people’s attention on an issue, and it’s just difficult to see a doorbell as a threat for the average person.
You know how when you finish prompting some code generator to build something, and you look over what it has built and feel a sense of emptiness even if it does what you want? I think about what I wish the prototype looked like, and basically start describing details that I expect to exist (think longer versions of e.g. “this should be using our internal graph library, and I figure we can model this task as a traversal, how far have you strayed from this and why?”) and let the agent analyze what it built against my expectations. I’ve spent hours in conversation just “refining the context” this way, and then I channel that into an update process. I figure the prototype is just about proving out behavior, and this next phase is about refining it into the pieces I’ll use elsewhere. It’s kinda fun, I’d absolutely burn out a coworker if I grilled their PRs the way I roast AI contributions :P
I’m still pretty skeptical about OpenRouter. I have a client implemented for them so I can use them with my harnesses, but at the same time that client was generated and tested in an hour or so just like all of the other llm provider clients that I have. Using these services interchangeably by just swapping out clients has so far been working well for me. I think when it comes down to it, the only real inconvenience that they’re solving is where I put my credit card number. Is there something key that I’m missing about this service (besides it being a nexus of attention) that warrants this kind of investment? Or is this truly the bar for starting a successful AI company :P
I once had someone start _arguing with me_ about stuff using generative text on Slack or generated email replies. Not even to provide information, but just to write flat denials to even continue discussion on the subject. This person had a very distinctive writing style, and the shift to the AI writing style felt pretty obvious and uncanny.
I can’t describe how disturbing it was to realize that my voice suddenly no longer mattered, and that I was speaking to something that would never get tired of creatively dismissing my ideas without ever really addressing them. This behavior compounded and was unaddressed by anyone, no one I talked to seemed willing to try actually pushing back against it. Best solution they had was to have physical meetings w/ n>1 people on each side in the room. Trust plummeted, and eventually all meetings with that team were recorded and transcripted, and people started talking like they were on stage vs trying to solve shared problems. Work ground to a halt on even basic things, and I ended up leaving. This was on a pretty major project that has a name people here would know, but don’t ask me what!
It has been funny to watch people’s attitudes on copyright change ever since ChatGPT blew up. All I used to hear and experience was copyright used by corporations to shut down open source projects threatening their business models, but now it is the savior of the little guy who is a victim of flagrant corporate violators. In the background, the wealthy and powerful disregard all of this and seem to do whatever they want, and the little guy looks at millions of dollars in legal costs to defend themselves in either case. Costs that are increasingly a rounding error to their opposition as they continue to grow by exploiting a broken system, and the “little guy” now includes whole industries.
I feel like adversarial interoperability more than free market capitalism should have been the death knell for most of the negatives highlighted in this post. Everyone is still so determined to make money from mere ideas however that we still use 1700s law designed to protect book publishers to enable the existence of “businesses” so warped in valuation that they are now trillion dollar entities yet always face the existential threat of copy+paste. What if the more profound truth is that tech is beneficial to humanity but inherently worthless to sell, and that our present woe’s shape is determined by the antiquated institutions built service this illusion of value? In an inevitable future age of generative AI as an accessible technology, as opposed to a business model with a moat, what even is our goal for such institutions? What sorts of creativity do we want motivate, and what meaningful regulatory constraints even are there to begin with? I hope we figure it out soon, because IP will be impossible to enforce post-deglobalization in any case.
Think it's just the hypocrisy. Either copyright for everybody or copyright for nobody is much more defensible than the current state of affairs, where infringing copyright is legal as long as you're rich. Some random guy in Nebraska had to pay $250,000 to a music company for downloading one MP3, but OpenAI can download all music that ever existed and pay nothing. Meanwhile they prosecute "Anna" who did the exact same thing, because "Anna" isn't politically well-connected.
> where infringing copyright is legal as long as you're rich.
This isn’t true. A rich person and a poor person can train LLMs on copyrighted material in 2026. How they acquired those materials matters. Wealthy corporations hold no legal advantage in this space. For example, Anthropic recently settled for $1.5 billion due to acquiring books via piracy: https://www.nytimes.com/2025/09/05/technology/anthropic-sett...
My understanding is that an individual could likely pirate the same books without paying a dime (not due to differing legal standards but simply due to the fact it would be hard to identify them in many jurisdictions). In a practical sense it seems corporations are held to a higher standard in this regard.
The discrepancy is that some people equate training a model with piracy even though they are not the same thing. This is typically due to intellectual laziness (refusal to understand the differences) or willful misrepresentation (due to being an ideologically opposed to generative AI). No need to make such a mistake here though.
Of course it's not the same thing -- it's way worse.
The piracy comes first, and it's exactly the same thing. GenAI Corp. can't train models on illicitly obtained media before illicitly obtaining said media. And that very thing is already what private individuals got and get sued for millions over.
The GenAI Corp., having gotten away with that unpunished, then goes on to commit further violations by commercially exploiting the media with neither a license to do so, nor any intentions to pay the rights-holders for their use.
By the media conglomerates' own math, these GenAI companies should all be drowning in lawsuits over kazillions of bajillions of dollars.
> The piracy comes first, and it's exactly the same thing. GenAI Corp. can't train models on illicitly obtained media before illicitly obtaining said media.
My contention is that this is not happening. Most generative AI companies do not source their training data from illegal torrents and the few that do are currently paying for it. Further, I suspect the companies that get away with it today are _smaller_ not larger.
Training data is typically sourced by scraping the publicly available web.
> Of course it's not the same thing -- it's way worse.
Setting aside your own moral standards here, we should at least be able to agree that from a legal standpoint training a model is not copyright infringement.
> A rich person and a poor person can train LLMs on copyrighted material in 2026.
Updating an old adage for the modern age:
“The law, in its majestic equality, forbids rich and poor alike to sleep under bridges, to beg in the streets, and to steal their bread.”
― Anatole France
As others have said, it's not a change. There's no inconsistency in applying copyright to protect people. When Gigantic Company uses copyright to bully the little guy who isn't doing anything to materially harm Gigantic Company, that's bad. When AI steals the little guy's work, that's bad. They're both bad. That's consistent. It's also obvious that it's consistent - i.e. I don't believe people making the "AI copyright complaints are funny" quip are being honest. I believe they are simply engaging in petty social politics.
>It has been funny to watch people’s attitudes on copyright change ever since ChatGPT blew up. All I used to hear and experience was copyright used by corporations to shut down open source projects threatening their business models, but now it is the savior of the little guy who is a victim of flagrant corporate violators.
I agree. My point in short is that we seem to reflexively frame right and wrong on an axis defined by copyright, and somehow we’ve lost sight of the fact that the law itself is used much differently than we might otherwise want.
Technolibertarians confuse free market capitalism via copyright-enabled businesses as a viable strategy for individual freedom, and we find with time that only bastards win in a competition with loose rules and high stakes. Those concerned for the continued flourishing of human creativity in the face of LLMs confuse copyright as a means for small creators to have some ownership over their work, when it actually just seems to be a cudgel that can only be wielded by the wealthiest. Same losing fight, different flavor. I ask: why do we continue to allow “ownership of ideas” to underlie the moral basis of our conversations to begin with?
I think it's more that we see copyright as a necessary evil that can be used to defend our rights, but will be abused by the powerful, regardless.
To me, the biggest sin of cyberlibertarianism is the assumption that "cyberspace" is de facto another universe, separate from material reality, that doesn't need to be affected by the mundane and vulgar rules of "meatspace." John Barlow refers to "your governments" as if using a computer actually separates him from the state in some meaningful way, as if he has ascended beyond the flesh and now looks down upon the world as a being of pure Mind. But of course, "cyberspace" is just computers, servers, infrastructure using power and resources and thus is inextricably subject to government and systems of law. Zion was never an escape.
So yes, because cyberspace doesn't actually change the rules of the game, we have to play the game, crooked as it is, with the hand we're dealt. The legal pretense of ownership and copyright is all we have. If you want to abandon the idea of "ownership" altogether, then the wealthiest and most powerful still wind up controlling everything by virtue of their wealth and power. What do you suggest?
The whole thing just shows a huge lack of imagination, at least for something which is supposedly a 'founding document'. Barlow's "cyberspace" is for irrelevant shit like furry larping or talking about the latest Deep Space 9. Its not a place where you do banking (or even watch DS9).
> John Barlow refers to "your governments" as if using a computer actually separates him from the state in some meaningful way, as if he has ascended beyond the flesh and now looks down upon the world as a being of pure Mind. But of course, "cyberspace" is just computers, servers, infrastructure using power and resources and thus is inextricably subject to government and systems of law. Zion was never an escape.
I don't understand what you're trying to say here, is it that "cyberspace" couldn't exist as anything "real" because governments can just shut down servers? That's why you can't buy drugs and credit card numbers online anymore, right? Sarcasm aside, you seem to be using the fallibility of the current-popular physical layer to dismiss the otherwise separate tangible "space" that does seem to exist when lots of people can communicate fluidly with each other across vast distances. Or is your critique centered on the ability of "cyberspace" to go beyond just communication and serve as a space one can actually "live" in?
> The legal pretense of ownership and copyright is all we have. If you want to abandon the idea of "ownership" altogether, then the wealthiest and most powerful still wind up controlling everything by virtue of their wealth and power.
Limiting abandonment of "ownership" to only "copyright" and IP generally, what do you propose the wealthy would control that would allow them to replicate present circumstances in "cyberspace"? The best I can think of would be communications infrastructure, and they didn't build that by themselves (at least in the US) to begin with.
For example, why would TikTok continue to be usable as a brainrot generator & propaganda tool when content is necessarily separate from the algorithm and presentation layers? Current bastards exploit their centralized control based on this house of cards ownership structure. Nothing is practically stopping users from cloning the contents from the cdn and writing a new frontend besides legal threats. This is true of almost every tech business that exists, and many of them themselves exploited this asymmetry during their founding. They exist because billionaires use the legal system to scare individual upstarts from threatening their business model.
> It has been funny to watch people’s attitudes on copyright change ever since ChatGPT blew up.
I doubt many individuals actually changed their opinions. Just that a large crowd of previously-silent people decided AI is a threat to them and they can attack it on copyright grounds. The AI revolution is a great argument against copyright law. The US's lax enforcement means that the incredible, world-changing tech could be built before the luddites got organised to try and stop it. The productive path appears to be illegal, but they took it anyway and we're all the better off for it.
Being rational and correct is a low bar, it is likely that all sides of any given debate a rational and correct to some extent. Someone dumping their entire life savings into a casino can be said to be rational and correct if the person doing it genuinely prioritises short term pleasure enough - still a stupid thing to do.
The reasons that jump out at me are that, as a society, we're setting up to produce a more stuff with less effort, provide higher quality advice to everyone at an absurdly low cost, revolutionise research and it looks like we're going to be able to get a step-change improvement in the quality of economic management which is huge in and of itself. The wins seem like they're going to be big.
> we're setting up to produce a more stuff with less effort
According to Jevon's paradox[0], this would lead to more consumption of resources. We're already straining at the limits of the Earth. Depletion and collapse won't be good for anyone.
> provide higher quality advice to everyone at an absurdly low cost
Given every LLM's propensity to hallucinate, the only quality advice is that which can be followed back to a human expert-vetted source. But we already have people who don't check sources and get bad advice.
> revolutionise research
Maybe, but AI is also being used in a mass spread of misinformation.
> a step-change improvement in the quality of economic management
I don't know exactly what you mean by this, but from what I'm seeing so far, this looks like it will massively increase wealth disparity, which is bad for most people.
This is to say: the autoregressive decoder-only transformer llm architecture as pioneered by openai is wildly simple for how revolutionary its results are. I was reading about non-learned classical SLAM systems (uses video + handcrafted math to produce 3d mappings of physical spaces while also locating the camera in those spaces) at the time, and comparatively speaking I’d say the math is about as complicated as ONE of the components in those complex formulations. The only reason frontier LLMs need 6-figure computers to run is because the model designers made the middle bit in those models REALLY BIG, dimensionally speaking. They just took the steam engine, made a few gargantuan versions of it, and are selling them as the ultimate source of power.
This was openai’s entire breakthrough. Making this particular model architecture larger leads to emergent capabilities like being able to pick the best ending to a story/set of instructions or answer questions about broad factual knowledge. I’ve been meanwhile watching these AI companies attempt, successfully, to sell this capability as some sort of robot consciousness hand-crafted by supergeniuses. The fact that they are getting away with it is almost as shocking to me as the discovery itself.
reply