Apologies if it sounds like promotional material, but the topic is something that has been discussed also here a lot. I also predicted a while back that vetting, curation, trust chains, and alike will be coming.
It affects science too (and there you'd want solid archiving as much as possible). Increasingly, meta-data is full of errors and general purpose search engines for science are breaking down, including even things like Google Scholar. I suppose some big science publishers are blocking AI bots too.
Did Google ruin it, or did advesarial activity between Google's algorithm and SEO ruin it? The latter seems more likely because the incentives make sense, and also inevitable.
It was. Advertising is incompatible with accurate data retrieval/routing. We've also implemented "obligation to deindex". So providing an unbiased index of the web as she is is essentially (in the U.S.) verboten.
> I suppose some big science publishers are blocking AI bots too.
That's a travesty, considering that a huge chunk of science is public-funded; the public is being denied the benefits of what they're paying for, essentially.
So the solution is to allow the AI scraping and hide the content, with significantly reduced fidelity and accuracy and not in the original representation, in some language model?
If it's publicly funded, why shouldn't AI crawlers have access to that data? Presumably those creating the AI crawlers paid taxes that paid for the science.
> If I build a business based off of consumption of publicly funded data, and that’s okay, why isn’t it okay for AI?
Because when you build it you aren't, presumably, polling their servers every fifteen minutes for the entire corpus. AI scrapers are currently incredibly impolite.
Plenty of public funded data isn't made free and public access. Sometimes you need to pay, or get a license, etc depending on what you're doing with it.
If anyone wants the surreal experience of seeing blogs and websites made by real humans they should check out https://marginalia-search.com
It's far from perfect but it does achieve its stated goal: of resurfacing real people on the internet.
It recently got some NLNet funding and I hope to see it flourish - to my knowledge there aren't any other projects trying to claw back control of the internet towards the commons.
I cannot say how valid it is but it is interesting because everyone else is saying the contrary (i.e., according to his data, the volume has gone quite rapidly down post-2021).
Oh no, people are mean to me :-O. I mean, sure, we can talk about sociopaths and whatnot, who may or may not have their means, but do we need such role models? Ends, not means. Kant?
reply