Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You say this like robots.txt doesn't exist.


it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?

implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”


so many things flat out ignore it in 2026 let's be real




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: