View Full Version : one robots.txt shared by multiple domains - needs to be different for each
My company has several sites that are all very similar, sharing the same set of local files on our machine -- including robots.txt. Up until recently, the robots.txt files could all be the same, but now they have to be different for each domain. We're running on an IIS server that we have full control of.
Does anyone know of a way to do this?
or to make it more simple, we have one main domain that we want the engines to have full access to, but all other domains need to be blocked. But each domain shares the same set of files on our server, so how do we accomplish the goal?
thanks in advance if anyone has solved this problem :)
nielsencl
10-22-2007, 03:36 PM
Sorry no one got back to you before now...
First you need to know that Robots.txt will only keep out those spiders that follow it's instructions. It will not keep your information from being found. To do that you will need to do much more that just use the robots.txt file.
Second, what I would suggest is to configure your IIS server to process .txt files as PHP or ASP files.
Once you do that, you can add programming code look at what site is being referred to and show the correct robots.txt information for that site.
To be safe, I would put the content that you don't want indexed on another server.
AussieWebmaster
10-22-2007, 06:05 PM
similar robots.txt files will not hurt you unless it is wrong!
nielsencl
10-22-2007, 07:50 PM
First of all, how would putting the content on another server protect it from being indexed? What would be different on the different server?
Secondly, all the sites share the same single robots.txt file. The user wants to use the one file to block spiders from all the sites except one.
The information I offered should provide one possible solution, but there may be others.
AussieWebmaster
10-22-2007, 08:13 PM
then it better be redirected... though why not have one each and avoid possible errors
Sure nielsencl, there are a number of ways to do complete this task.
When folks are sharing a robots.txt and confidential information is involved, I normally suggest another box just to reduce the chance of information being indexed in the main SERPS due to future changes in robots.txt and/or content (kind of "idiot proofing"). Also, by having your confidential content on another server it's easy to set-up error free "enterprise search" for that content with limited access.
As Aussie pointed out, there would need to be a redirect but why not have one each? Having one each, would be the best way to avoid the need for redirects as well as all of the other possible issues both now and in the future.