Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
SR2Z
on Jan 18, 2025
|
parent
|
context
|
favorite
| on:
Amazon's AI crawler is making my Git server unstab...
IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
prinny_
on Jan 18, 2025
|
next
[–]
Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.
compootr
on Jan 18, 2025
|
parent
|
next
[–]
OAI is using others' work to resell it in models. IA uses it to presrrve the history of the web
there is a case to be made about the value of the traffic you'll get from oai search though...
SR2Z
on Jan 24, 2025
|
parent
|
prev
|
next
[–]
It does depend a lot on how you feel about IA's integrity :P
amarcheschi
on Jan 18, 2025
|
prev
[–]
I also don't think they hit servers repeatedly so much
Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
