Web development Birmingham

How to create a domain / subdomain specific robots.txt file

By Andrew Ward By Andrew Ward Jun 05, 2014 web development, seo

This is how to create a robots.txt file that only appears under selected domain name / subdomain names. A common case where this may be needed is if you are setting up a testing domain name whilst a site is being built and you want to avoid the scenario where you forget to alter the file and accidentally deny the website from being seen by search engines.

Firstly create a robots.txt file as normal but give it a slightly different name. In my case mine is called "robots-staging.txt":

User-agent: *
Disallow: /

Then you need to add a htaccess rule (I.e. you need to edit your .htaccess file) that for the domain in question re-writes the file name to appear as robots.txt so that it can be picked up by search engine spiders. In this case "testdomain.scorchsoft.com" would be the one that we would like the subdomain to appear under:

RewriteEngine on

RewriteCond %{HTTP_HOST} ^www.testdomain.scorchsoft.com$
RewriteRule ^robots\.txt$ robots-staging.txt
RewriteCond %{HTTP_HOST} ^testdomain.scorchsoft.com$
RewriteRule ^robots\.txt$ robots-staging.txt

And that is it, the robots.txt will now appear for the domain "testdomain.scorchsoft.com" but will not show up for any other domain pointed to the hosting in question.

Did you find this article useful? Sharing is a great way to show support, and ensures that we can continue to create more quality articles:

Get a free quote