It depends on the model, but from my experiments (quantizing one layer of a model to 2-bit and then training the model with that layer in 2-bit to fix the damage) the first layer is the most sensitive, and yes, the last layer is also sensitive too. The middle layers take the best to quantization.
Different components of a layer also have a different sensitivity; e.g. the MLP downscale block damages the model the most when quantized, while quantizing the Q projection in self attention damages the model the least.
Do you think this is going to stop anyone, considering everyone is already training on All Rights Reserved content which is inherently more restrictive than whatever license you're going to use?
There's no story. You need to remember - big corporations are not your friend. They're your enemy. They don't care about you. They don't care about doing good. They care about money. They care about control. They care about their stock price. That's it.
You might ask - but what about the people who work at those corporations? And that's also pretty simply explained by this classic quote: it is difficult to get a man to understand something, when his salary depends on his not understanding it.
They are not your enemy either. They are… businesses. Whose purpose is to survive and thrive. Just because you don’t like them or what they do doesn’t make them your enemy. And, lots of very talented and smart people work there every day for their own personal reasons. No need to bash or show hatred to them.
Less of the "not liking" and more of an "are an existential threat to."
This isn't high school. This is about real people having real experiences of fear, stress, violence, and horror facilitated by deliberate cultural engineering.
If the very talented and smart people don't get that, that's a them problem.
What a sad, sad take. Do you even know what the word “enemy” means? Just because I don’t like my neighbor doesn’t make them my enemy. We are not going to war with each other, we just don’t like each other’s company. Just because I don’t like your comments on HN doesn’t mean I hate you. Good grief.
You've entered into a logical fallacy there - the parent was saying that not liking someone or what they do is a prerequisite for them becoming an enemy. They did NOT say that everyone you don't like is your enemy, which is the straw man you chose to respond to.
I disagree. If you take statement X to be "you don't like them" and Y to be "they're your enemy". Then OP said "Just because X is true, it doesn't mean Y is true". In other words, "X does not imply Y". Meneth said "yes it does". In other words, "X implies Y".
All enemies are people you dislike and / or people who do things you don't like. This does not make the opposite true, not all people you dislike or who do things you don't like are your enemy. The statement "all cats are black" does not also mean "all black things are cats".
I roughly agree with you on that (with the caveat that e.g. opposing army generals can be enemies but admire and respect each other). I disagree that Meneth was saying what you said.
But if your neighbour actively and deliberately makes your life worse then they certainly could be your enemy.
If I’m queer and Facebook is actively censoring queer content then that’s more significant to me than just being a difference of opinion. The company is actively suppressing my way of life.
Maybe the word “enemy” is too much but if so I think describing the idea as “sad” is equally as so. Giving a corporation a pass on behaviour you consider abhorrent simply because it’s a company and not a person seems pretty sad to me.
>If I’m queer and Facebook is actively censoring queer content then that’s more significant to me than just being a difference of opinion. The company is actively suppressing my way of life.
Why queer community will not find an alternative app?
This is the incredibly profitable contradiction Facebook lives in.
They do everything they can to become the central place for online communication and profit enormously from that. But they reject any of the responsibility that ought to come along with that, the refrain being what you're saying here: "well, you can always just go somewhere else"
Except that when online communication is as deeply siloed as it is it's extremely difficult to set up an alternative. How will people even find out about it when their entire online lives are lived on Facebook? This capture is exactly what Meta wants. Remember internet.org?
No, because then what happens when the place they move to starts censoring them as well? Then all the places start censoring them? You're basically arguing for "separate but equal", and we know how that works out. The correct move is to fight for your rights, not to assuage bigotry.
And you are arguing every business must support your agenda, and if not, they are your "enemy"? What an odd take. Again, you are free to use other means of social media to spread your message but no one is obligated to read or support it. And, that does not make them the enemy.
You already said that. It does not answer the question. Moving to another app doesn't solve anything, because we still haven't answered the question of why they should have had to move in the first place! It's the same situation if they move to a new app, nothing has changed.
At this point we have gone in a circle, I must assume I won't get a genuine answer to the only thing I have asked despite trying to engage genuinely in conversation. Have a good day.
And by your own logic, how does censoring content actively suppress your way of life? Did someone from Meta go to your place of residence and actively threaten to harm you? Sure, maybe you don’t like the censorship, but how does that make them your enemy? Have you openly declared war on them? If you don’t like their content, simply move along.
> And by your own logic, how does censoring content actively suppress your way of life?
Because it erases our existence, which is what a substantial slice of straight society wants. Queer content and spaces are important for queer adults, because it gives us places to comfortably be ourselves without feeling subject to leering or judgement from bigots, and safety in numbers in case someone starts something. It gives us people to be among who we can talk to, form community with, and support one another. And for people just coming up, it's literally lifesaving. Numerous studies have shown that queer-leaning teens and kids are MUCH safer when they have access to safe places to explore who they are, even if they don't "turn out" that way, prevents awful, irreversible things. [1,2,3] Not to mention it can be lifesaving also when their parents are bigots themselves and they need a way out.
> Sure, maybe you don’t like the censorship, but how does that make them your enemy?
The bridge between "they suppress expressions of who I am" and "they participate in my extermination" has been proven to be quite short and easily traversed for queers many times, and for racial groups, and for religious groups too. [4]
By your definition they may not be my enemy today, but they may be in a short period of time.
> If you don’t like their content, simply move along.
This is actually great advice for people who keep trying to take down queer content.
Edit: And this is exactly what figures like Breitbart have been openly trying to do for over a decade. And it isn't just him either, you have the Family Research Council, Fox News hosts, Daily Wire personalities like Matt Walsh, and Libs of TikTok have all made careers out of normalizing queer erasure. For them, "winning the culture war" is not only their stated, in-text goal, it's a means of pushing us out of public life: sometimes just running us out of town, other times things too ugly to say aloud.
Erases your existence? Would your existence be threatened if Meta was not a company? What about the countless number of other companies who are not pushing your content? Do you feel threatened by them? Now I see why you chose the word "hate"...
No, my existence isn’t contingent on Meta existing. But when a platform with billions of users decides queer content is unwelcome, it erases us from one of the largest public squares in the world, at a time when public squares are at a premium. That’s not the same as "some random company doesn’t carry my stuff."
There's also a difference between not amplifying something and actively suppressing it. Neutral omission is one thing; deliberate censorship is another. When queer content is singled out for removal, it sends a message: you don’t belong here. That's erasure.
History shows us that erasure is rarely neutral. It's part of a continuum: silence leads to exclusion leads to violence. Pretending censorship is harmless ignores the fact that queer people have lived through this cycle many times before, and we're far from alone.
> And, lots of very talented and smart people work there every day for their own personal reasons. No need to bash or show hatred to them.
Lots of very talented and smart people work for big tobacco, Aramco, Stake (crypto gambling), Kick (streaming of crypto gambling), Purdue (made billions on manufacturing an opioid epidemic), DuPont, Shein, Nestlé, NSO group, the GEO group (private prison industry), Clearview (facial recognition at scale including for ICE) and indeed Meta.
What would be a resaonable cause to bash them, in your view, if not disliking what they do?
I don't think we should hate them or show them hatred. I don't think that if you're working at a company that's suppressing someone's way of life you're somehow above criticism or contempt.
The question is, why do you feel the need to bash them? Do you feel the need to bash the coders of YouTube because they have ads or censor content? Do you feel the need to say ugly things to your grocery store because they don’t actively have the goods you want? Are they your enemy because they hire a certain type of person?
I don't mean to be a dick, but no, the question was what is a reasonable cause to bash someone if it's not disliking what they do. I don't know if these weird Socratic replies are meant to be thoughtful but they read as dismissive and condescending.
But hey, I can also play stupid games!
> The question is, why do you feel the need to bash them?
Why do you feel the need to defend them? (I answer this question less flippantly below.)
> Do you feel the need to say ugly things to your grocery store because they don’t actively have the goods you want?
Is not stocking garbanzo beans censorship? Why do you think this is equivalent?
> Do you feel the need to bash the coders of YouTube because they have ads or censor content?
Depends what Youtube is advertising. Depends what they're censoring.
> Are they your enemy because they hire a certain type of person?
Who'd they hire?
...
There is a difference between my grocery store stocking or not stocking something and having problems with a multi-national trillion dollar company that has wedged itself itself into most people's daily lives and has a history of censoring content to curry favour with authoritarians.
I sympathize with the folks who are working there trying to change things for the better, and I sympathize with the people who are legitimately stuck for whatever reason (don't know a lot about visas to the US, but those are probably a good reason). I also think they're tough enough to take it when they dunked on, and have the reading comprehension to realize that when people are critical of Facebook employees, there's context where it absolutely makes sense. Being a Facebook employee is not an identity, it's just a job. Facebook has pivoted to censoring queer content at a time when queer people are being marginalized after years of gains. Most of my ire is directed at the executives and management, but yeah, if you work at Facebook knowing what they do, you're not getting a pass.
My replies are not meant to be dismissive and condescending - they are just frank/honest questions. No need to try to decipher a hidden message.
BTW - Meta wedged itself into most people's lives because the people let it happen. It started off well enough, but just like many companies, they adjusted their content based on the people consuming the platform. Its (Meta's) survival is based on getting views and posting ads. That's the business model. If they started showing content that appealed to a small percentage of their viewership, they would probably go out of business.
While privately owned business can put ethics before outright profit, large public companies are always bound to become assholes so yes they become de facto our enemies while also being our best friends because our pension depends on them.
Some businesses go out of their way to be awful and they deserve every ounce of scorn. Some even know they are causing harm, bury the evidence and continue.
Also, simping for these companies is such a bad look.
I dunno. I used Market Place yesterday to get a new dry erase board (new to me). And, many people use FB to communicate with friends and family. How is that working to make the world a worse place?
No idea. There was once this German guy who planted flowers in front of factories. His name was Adolf Hitler and I'm not sure why everyone hates him so much. (This is sarcastic and do you see the problem with your comment now?)
> And that's also pretty simply explained by this classic quote: it is difficult to get a man to understand something, when his salary depends on his not understanding it.
I've come to strongly dislike this quote, because it's so often used on HN to decide that whoever's disagreeing with you is doing it for simple, stupid and greedy reasons, thus absolving you of the duty to think a bit about whether there might be nuance you're missing.
We all have limited cognitive budget. Keep nuance investment were there is relevancy for it makes more sense than spending it on things where it’s not going to bring any value to anyone.
> Open source is what it is today because it's built by people with a spine who stand tall for their ideals even if it means less money, less industry recognition, lots of unglorious work and lots of other negatives.
With all due respect, don't you see the irony in saying "people with a spine who stand tall for their ideals", and then arguing that attaching "restrictions" which only affect the richest megacorporations in the world somehow makes the license not permissive anymore?
What ideals are those exactly? So that megacorporations have the right to use the software without restrictions? And why should we care about that?
Anyone can use the code for whatever purpose they want, in any way they want. I've never been a "rich megacorporation", but I have gone from having zero money to having enough money, and I still think the very same thing about the code I myself release as I did from the beginning, it should be free to be used by anyone, for any purpose.
> I do not mind having a license like that, my gripe is with using the terms "permissive" and "open source" like that because such use dilutes them. I cannot think of any reason to do that aside from trying to dilute the term (especially when some laws, like the EU AI Act, are less restrictive when it comes to open source AIs specifically).
Good. In this case, let it be diluted! These extra "restrictions" don't affect normal people at all, and won't even affect any small/medium businesses. I couldn't care less that the term is "diluted" and that makes it harder for those poor, poor megacorporations. They swim in money already, they can deal with it.
We can discuss the exact threshold, but as long as these "restrictions" are so extreme that they only affect huge megacorporations, this is still "permissive" in my book. I will gladly die on this hill.
> Good. In this case, let it be diluted! These extra "restrictions" don't affect normal people at all,
Yes, they do, and the only reason for using the term “open source” for things whose licensing terms flagrantly defy the Open Source definition is to falsely sell the idea that using the code carries the benefits that are tied to the combination of features that are in the definition and which are lost with only a subset of those features. The freedom to use the software in commercial services is particularly important to end-users that are not interested in running their own services as a guarantee against lock-in and of whatever longevity they are able to pay to have provided even if the original creator later has interests that conflict with offering the software as a commercial service.
If this deception wasn't important, there would be no incentive not to use the more honest “source available for limited uses” description.
> I couldn't care less that the term is "diluted" and that makes it harder
It also makes life harder for individuals and small companies, because this is not Open Source. It's incompatible with Open Source, it can't be reused in other Open Source projects.
Terms have meanings. This is not Open Source, and it will never be Open Source.
> It also makes life harder for individuals and small companies, because this is not Open Source. It's incompatible with Open Source, it can't be reused in other Open Source projects.
I'm amazed at the social engineering that the megacorps have done with the whole Open Source (TM) thing. They engineered a whole generation of engineers to advocate not in their own self-interest, nor for the interest of the little people, but instead for the interest of the megacorps.
As soon as there is even the tiniest of restrictions, one which doesn't affect anyone besides a bunch of richiest corporations in the world, a bunch of people immediately come out of the woodwork, shout "but it's not open source!" and start bullying everyone else to change their language. Because if you even so much as inconvenience a megacorporation even a little bit it's not Open Source (TM) anymore.
If we're talking about ideals then this is something I find unsettling and dystopian.
I hard disagree with your "It also makes life harder for individuals and small companies" statement. It's the opposite. It gives them a competitive advantage vs megacorps, however small it may be.
Nobody cares if they use a license that inconveniences megacorporations. The issue is how they try to present the license.
> start bullying everyone else to change their language
Either words matter or they do not. If words matter, then trying to dilute the term is a bad thing because it tries to weaken something that matters. If words do not matter, then the people who "bully everyone" can be easily ignored. You cannot have these two things at the same time.
> 10-13 minutes if I remember correctly from booting the game to actually being able to do anything besides mash buttons to try and skip the cutscenes.
Genuinely curious - if you don't care about the story then why play an RPG? When you're speedrunning - sure, skip all of the cutscenes, but when you're playing casually - why would you want to do that?
> This is especially damning when the long unskippable cutscene is during a boss fight or something which you might fail afterwards and cannot save.
Some games have started to get this right, either by making cutscenes you've seen skippable, or by just automatically skipping straight to the battle if you've already been through it once. I suspect one reason it didn't happen on older games was the need to explicitly save, rather than autosaving.
Unlike a movie, when done well, the combat/grinding add to player engagement because it places the player in direct charge of the characters' growth from a nobody to a legend. You can't get that from a movie.
I agree with your take here that he should care about the cut scenes/story if bothering to play, but this has gotten especially bad in newer games where they try to shove you right into the game before you can tweak settings. I never played through Bravely Default on 3DS because the opening scene used the English dub instead of the original audio, and I had to skip it to access the settings and change languages, then there was no way to rewatch that opening scene. I've similarly avoided their other games like Octopath Traveler as I suspect they have the same issue. It seems like an accessibility issue. I don't think they should ever stop you from getting to the settings first thing. I am not entertained by them trying to be overly cinematic. I don't think it would kill them to wait until you hit "start new game".
Starting off with 10min of exposition is too much and it’s lazy. You don’t even know if you’re going to like the game yet. Do some en media res story telling and get on with it.
Most games I don’t care about the deep exposition. I’m fine with a vague notion and then starting from the main character’s insertion into it where the gameplay starts.
10 minutes is a long time now? Is this what TikTok does to a person’s brain? Cartoon Network typically had 11 minute episodes, so you’re complaining that you don’t even have the attention span of a child.
I play RPGs for the fun of turning time and grind into more advanced abilities (eg going from getting slaughtered by dragons in Skyrim to being the one doing the slaughtering).
There are few games where the story has mattered to me, and even basically no games where the cutscenes did.
Edit: the presence of story and cutscenes in a game I enjoy is basically correlation and not causation (for me).
> It's incredibly clear who the devs assume the target market is.
Not "assume". That's what the target market is. Take a look at civitai and see what kind of images people generate and what LoRAs they train (just be sure to be logged in and disable all of the NSFW filters in the options).
Yep. It's pretty difficult to fine tune, mostly because it's a distilled model. You can fine tune it a little bit, but it will quickly collapse and start producing garbage, even though fundamentally it should have been an easier architecture to fine-tune compared to SDXL (since it uses the much more modern flow matching paradigm).
I think that's probably the reason why we never really got any good anime Flux models (at least not as good as they were for SDXL). You just don't have enough leeway to be able to train the model for long enough to make the model great for a domain it's currently suboptimal for without completely collapsing it.
So what this does - you trigger the model once with a negative prompt (which can be empty) to get the "starting point" for the prediction, and then you run the model again with a positive prompt to get the direction in which you want to go, and then you combine them.
So, for example, let's assume your positive prompt is "dog", and your negative prompt is empty. So triggering the model with your empty prompt with generate a "neutral" latent, and then you nudge it into the direction of your positive prompt, in the direction of a "dog". And you do this for 20 steps, and you get an image of a dog.
The guidance here was distilled into the model. It's cheaper to do inference with, but now we can't really train the model too much without destroying this embedded guidance (the model will just forget it and collapse).
There's also an issue of training dynamics. We don't know exactly how they trained their models, so it's impossible for us to jerry rig our training runs in a similar way. And if you don't match the original training dynamics when finetuning it also negatively affects the model.
So you might ask here - what if we just train the model for a really long time - will it be able to recover? And the answer is - yes, but at this point the most of the original model will essentially be overwritten. People actually did this for Flux Schnell, but you need way more resources to pull it off and the results can be disappointing: https://huggingface.co/lodestones/Chroma
Thanks for the extended reply, very illuminating. So the core issue is how they distilled it, ie that they "baked in the offset" so to speak.
I did try Chroma and I was quite disappointed, what I got out looked nowhere near as good as what was advertised. Now I have a better understanding why.
> How much would it cost the community to pretrain something with a more modern architecture?
Quite a lot. Search for "Chroma" (which was a partial-ish retraining of Flux Schnell) or Pony (which was a partial-ish retraining of SDXL). You're probably looking at a cost of at least tens of thousands or even hundred of thousands of dollars. Even bigger SDXL community finetunes like bigASP cost thousands.
And it's not only the compute that's the issue. You also need a ton of data. You need a big dataset, with millions of images, and you need it cleaned, filtered, and labeled.
And of course you need someone who knows what they're doing. Training these state-of-art models takes quite a bit of skill, especially since a lot of it is pretty much a black art.
> Search for "Chroma" (which was a partial-ish retraining of Flux Schnell)
Chroma is not simply a "partial-ish" retraining of Schnell, its a retraining of Schnell after rearchitecting part of the model (replacing a 3.3B parameter portion of the model with a 250M parameter replacement with different architecture.)
> You're probably looking at a cost of at least tens of thousands or even hundred of thousands of dollars.
For reference here, Chroma involved 105,000 hours of H100 GPU time [0]. Doing a quick search, $2/hr seems to be about the low end of pricing for H100 time per hour, so hundreds of thousands seems right for that model, and still probably lower for a base model from scratch.
If you like old computers and are interested in live coding music then I highly recommend to check out this video (Making 8-bit Music From Scratch at the Commodore 64 BASIC Prompt):
It depends on the model, but from my experiments (quantizing one layer of a model to 2-bit and then training the model with that layer in 2-bit to fix the damage) the first layer is the most sensitive, and yes, the last layer is also sensitive too. The middle layers take the best to quantization.
Different components of a layer also have a different sensitivity; e.g. the MLP downscale block damages the model the most when quantized, while quantizing the Q projection in self attention damages the model the least.
reply