LLMs create text. They don't understand them in the traditional sense.
LLMs are backwards. Much like how Stable Diffusion is text-to-image, not image-to-text. You need to rework the neural network entirely to do what you're asking.
-------
Its a neat trick that you can train LLMs to perhaps, ingest "text" followed by a question (ex: "Is the previous paragraph spam?"), and then the text-generator generates a yes-or-no response that is somewhat pleasing.
But its not quite the same as what you're asking for.
Have you... actually used the image recognition features of even something like the web-accessible ChatGPT 4 interface, or are you just guessing here?
I feed it unstructured images of all kinds of things and it generally does a rather good job of describing them.
Just now, I found (what is allegedly) a screenshot of a Facebook ad about microdosing mushrooms. I fed the image to the bot and asked it to identify whether or not it was an ad, and the content of it, and to rate the legality of it on a scale of 1 to 50, with 1 being perfectly legal and 50 being criminally punishable.
It nailed all three questions and gave it a score of 45.
I then did the same thing with a screenshot of a Facebook ad for an Audi Q5, and it nailed all three questions and gave it a score of 1.
I followed this up with some human-made drawing of 4 horses that I found fairly randomly. It also nailed all three questions and gave it a score of 1.
Satisfied with the result, I then stopped. I don't get paid enough to make a study of this. I have little doubt that these results are reproducible.
(And is this an LLM trick or something else? Perhaps a combination of things? Is it AI, LLM, LMNOP, or some kid in a sweatshop overseas? I don't really know -- that distinction is also beyond my pay grade.)
> (And is this an LLM trick or something else? Perhaps a combination of things? Is it AI, LLM, LMNOP, or some kid in a sweatshop overseas? I don't really know -- that distinction is also beyond my pay grade.)
That's the thing: The hidden layers that process the text _MUST_ have some kind of language structure getting figured out. Otherwise none of the behaviors of ChatGPT make sense.
Interpreting those layers and passing them to the user in some way should be the core focus IMO anyway.
On the other hand: there's also the problem of brute force. If they paid 50,000 cheap "Mechanical Turks" to manually teach an AI that the "top left" of an image is pixel coordinates (0,0), that's... trickery. Its no longer intelligence. Its disconnected and not innate to the model anymore.
The nature of the training set is what would determine how useful any of this actually is, and how well it generalizes to new tasks.
If I understand correctly, every LLM has an embedding function which can reduce a block of text to a coordinate in high-dimensional space. That coordinate can be used to determine how likely something is to be talking about selling illegal drugs and automatically flag posts for manual review.
But given that the problem seems to be in the "manual review" because users constantly report these and get brushed off, that "neat trick" you wrote about seems like it could really help explain to manual reviewers exactly why a particular ad violate's Meta's content guidelines.
> If I understand correctly, every LLM has an embedding function which can reduce a block of text to a coordinate in high-dimensional space. That coordinate can be used to determine how likely something is to be talking about selling illegal drugs and automatically flag posts for manual review.
Sounds like Word2Vec, which isn't necessarily an incompatible technique. I'm not exactly on the cutting edge of Language Models here.
I'm sure the information is sitting in the "hidden layer" somewhere. But LLMs have very many hidden layers. Its not something that a programmer can just "pluck" out of the network.
I use LLM embeddings all the time at my day job (and I work in a very, very low-tech "tech" job, basically an IT department). It's not even remotely difficult and works incredibly well for sifting through absurd amounts of data to filter it down to something that is easy to human-review.
> I think you may be commenting "confidently" on technologies that you're not very familiar with. It is something that we all struggle with sometimes.
Your snarkiness aside, what I've said earlier I stand by.
LLMs are a network of neurons / self-learning units that point towards the "next" word to be predicted". This is fundamental to the architecture of this entire system.
I recognize that there's some kind of parsing and understanding from tokens -> hidden layers of the network. That's just how the math works.
Perhaps you should educate yourself upon neural networks and the fundamental math at play here before caustically criticizing everyone who contributes to a discussion.
I admit that I don't know what an embedding is but I know it's something that is created by a neural network's output layer (or the post-processing of the output layer). This is once again, fundamental to the math of the neural nets at play here. So anything I know about output layers applies to whatever an embedding is.
---------
Not everyone in this field is working off of just using the damn thing. Some of us have experience in the layout and underlying theory of neural nets. And I'm confident in what I've said.
I’m sorry that I came across so different from how I meant to. I was intending to evoke a tone more similar to one I had recently here[0]. That is my fault and I should work on my communication.
> I admit that I don't know what an embedding is but I know it's something that is created by a neural network's output layer (or the post-processing of the output layer).
The embedding layer comprises the first layers of a transformer model, definitely not the “output layer”. It is typically trivially easy to isolate the output of the embedding layer because it generates the “actual” input to the LLM. The LLM has a certain number of input parameters (typically 784) and some length of text usually won’t fit precisely into 784 parameters (it will be longer or shorter than that). All the embedding layer does is map arbitrary length text to a single 784-dimension vector, which actually can be passed to the input of the transformer.
Ever since the paper “Attention is all you need”, the embedding layers have always been the first layers of a transformer to act on the text input. Most of the output is also sent back to the embedder, but this is only so that it can be used as input again to the “attention” layers.
That's exactly what I'm asking for. Yes, technically LLMs do generate text but with some prompt engineering, can be used to classify things, especially ones that require Natural Language Understanding. We have literally used them for highly targeted classification tasks and they've done a great job.
Image recognition too, of course (using Machine Learning). My point is that, all the stuff FB has spent billions on in the last few years, they have the ability to fix these issues in their sleep.
As others have said, the key is that regulators are absent.
LLMs are backwards. Much like how Stable Diffusion is text-to-image, not image-to-text. You need to rework the neural network entirely to do what you're asking.
-------
Its a neat trick that you can train LLMs to perhaps, ingest "text" followed by a question (ex: "Is the previous paragraph spam?"), and then the text-generator generates a yes-or-no response that is somewhat pleasing.
But its not quite the same as what you're asking for.