> Web admins are not trying to optimize their crawlers, they are trying to stop their crawlers breaking sites.
Actually they often do and that's one of the original purposed of robots.txt - to get search engines to stop wasting time on indexing worthless crap like endless dynamically generated pages. It's only relatively recently that most crawlers had a hostile relationship with website operators.
Actually they often do and that's one of the original purposed of robots.txt - to get search engines to stop wasting time on indexing worthless crap like endless dynamically generated pages. It's only relatively recently that most crawlers had a hostile relationship with website operators.