Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Good question - I am just putting up robots.txt, and seeing little to no decrease in traffic. I have not tried verifying that server logs user agent corresponds to specific IP addresses. Do you have resources where all the AI bots post their list of IP addresses? Would be easier to just ban by IP completely. From what I've read these bots rotate and use residential blocks so I am not sure I can even see all of them.


Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: