User-agent: * Crawl-delay:240 RewriteEngine on RewriteCond %{HTTPS} on RewriteRule ^robots.txt$ robots-https.txt