Robots.txt meant for search engines don’t work well for web archivesblog.archive.org3 pointsnathcd9 years ago