# See http://www.robotstxt.org/robotstxt.html for documentation # User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-Agent: XoviBot Crawl-Delay: 5