User-agent: * Disallow: Host: mysite.com User-agent: * Crawl-delay: 2 # timeout 2 sec.