Robots.txt meant for search engines don’t work well for web archivesblog.archive.org295 pointsr7219 years ago