Worpress Question

[savantcreative]

I am trying to conduct an experiment on a Wordpress installation and need some help. I would like to create the condition that when a visitor navigates to http://example.com/robots.txt they will be shown the contents of my blogs index page however it will be named http://example.com/robots.txt
I am on an Apache server and can use mode rewrite.
Thanks so much for your help.

[AussieWebmaster]

hard to do a redirect of the robots.txt file as the spiders need to see it

[SEORockStar]

I don't think this could be possible, as AussieWebmaster told the Spiders won't be able to see it

[savantcreative]

Apparently it is possible because I have seen it. I am wondering if it has something to do with keeping bad spiders out of your site?

[AussieWebmaster]

but what is the purpose of it? you would need to do some type of user recognition on the server to send all but spiders/bots to a different page when they are looking for domain.com/robots.txt

[paulseoworks]

My thoughts is that this looks like a dangerous experiment. one worth taking a shot at, but id only do that on New sites, not an existing one.

[AussieWebmaster]

cloaking is dangerous

[jesicawillss]

I think its not possible. And, clocking is not good.

[paulseoworks]

Agreed Aussie/Jess. Cloaking is bad stuff and it is surprising/shocking how many people still do it

[Dodger]

Originally Posted by savantcreative

Apparently it is possible because I have seen it. I am wondering if it has something to do with keeping bad spiders out of your site?

As all have said, what is the point?

You really don't want to keep spiders from reading robots.txt, or try to redirect them.

What you can do, is read your log files and do a block in .htaccess of the IP Class 3 addresses (I use class 3 to block entire blocks of 256 addresses ... assuming they are from same server).

You can block higher classes, or ranges of addresses, that are known to be from pesky bots. You should only do this if it is a REAL problem.

order allow,deny
deny from 192.168.
deny from 10.0.0.
allow from all

The above code denies access for a class 2 (192.168) and class 3 (10.0.0) address range. The later blocks 256 addresses, the former 256*256 addresses.

I usually add this when I get hit by scrapers. They can come in and hit you hard. You really can't go by their "user-agent" element, for it changes. Better to nip the IP address in the bud.