Robots.txt is a suicide notearchiveteam.org115 pointspanza15 years ago"Archive Team interprets ROBOTS.TXT as damage and temporary madness, and works around it. Everyone should. If you don't want people to have your data, don't put it online."