At very least robots.txt is from 1994; it has been part of the web almost from the start (web became public in 1991, so within 3 years).

Claiming ignorance here would be just a little bit disingenuous.