# Begin robots.txt User-agent: * Disallow: /cgi-bin/ Disallow: /cgibin/ Crawl-delay: 10 User-agent: msnbot Crawl-delay: 10 User-agent: Teoma Crawl-delay: 10 User-agent: Slurp Crawl-delay: 10