Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > CMS and SEO
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 01-04-2011   #1
savantcreative
Ethical Marketing Firm
 
Join Date: Mar 2008
Posts: 45
savantcreative is on a distinguished road
Worpress Question

I am trying to conduct an experiment on a Wordpress installation and need some help. I would like to create the condition that when a visitor navigates to http://example.com/robots.txt they will be shown the contents of my blogs index page however it will be named http://example.com/robots.txt
I am on an Apache server and can use mode rewrite.
Thanks so much for your help.
savantcreative is offline   Reply With Quote
Old 01-06-2011   #2
AussieWebmaster
Forums Editor, SearchEngineWatch
 
AussieWebmaster's Avatar
 
Join Date: Jun 2004
Location: NYC
Posts: 8,154
AussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant future
Re: Worpress Question

hard to do a redirect of the robots.txt file as the spiders need to see it
__________________
Bruce Gillmer - MTV EMA
AussieWebmaster is offline   Reply With Quote
Old 01-09-2011   #3
SEORockStar
Member
 
Join Date: Mar 2010
Posts: 314
SEORockStar is on a distinguished road
Re: Worpress Question

I don't think this could be possible, as AussieWebmaster told the Spiders won't be able to see it
SEORockStar is offline   Reply With Quote
Old 01-11-2011   #4
savantcreative
Ethical Marketing Firm
 
Join Date: Mar 2008
Posts: 45
savantcreative is on a distinguished road
Re: Worpress Question

Apparently it is possible because I have seen it. I am wondering if it has something to do with keeping bad spiders out of your site?
savantcreative is offline   Reply With Quote
Old 01-11-2011   #5
AussieWebmaster
Forums Editor, SearchEngineWatch
 
AussieWebmaster's Avatar
 
Join Date: Jun 2004
Location: NYC
Posts: 8,154
AussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant future
Re: Worpress Question

but what is the purpose of it? you would need to do some type of user recognition on the server to send all but spiders/bots to a different page when they are looking for domain.com/robots.txt
__________________
Bruce Gillmer - MTV EMA
AussieWebmaster is offline   Reply With Quote
Old 02-03-2011   #6
paulseoworks
 
Posts: n/a
Re: Worpress Question

My thoughts is that this looks like a dangerous experiment. one worth taking a shot at, but id only do that on New sites, not an existing one.
  Reply With Quote
Old 02-04-2011   #7
AussieWebmaster
Forums Editor, SearchEngineWatch
 
AussieWebmaster's Avatar
 
Join Date: Jun 2004
Location: NYC
Posts: 8,154
AussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant futureAussieWebmaster has a brilliant future
Re: Worpress Question

cloaking is dangerous
__________________
Bruce Gillmer - MTV EMA
AussieWebmaster is offline   Reply With Quote
Old 02-19-2011   #8
jesicawillss
Member
 
Join Date: Oct 2010
Posts: 118
jesicawillss is on a distinguished road
Re: Worpress Question

I think its not possible. And, clocking is not good.
jesicawillss is offline   Reply With Quote
Old 02-20-2011   #9
paulseoworks
 
Posts: n/a
Re: Worpress Question

Agreed Aussie/Jess. Cloaking is bad stuff and it is surprising/shocking how many people still do it
  Reply With Quote
Old 07-14-2012   #10
Dodger
Honorary Member
 
Dodger's Avatar
 
Join Date: Jun 2004
Location: Central US
Posts: 349
Dodger has a spectacular aura aboutDodger has a spectacular aura aboutDodger has a spectacular aura about
Re: Worpress Question

Quote:
Originally Posted by savantcreative View Post
Apparently it is possible because I have seen it. I am wondering if it has something to do with keeping bad spiders out of your site?
As all have said, what is the point?

You really don't want to keep spiders from reading robots.txt, or try to redirect them.

What you can do, is read your log files and do a block in .htaccess of the IP Class 3 addresses (I use class 3 to block entire blocks of 256 addresses ... assuming they are from same server).

You can block higher classes, or ranges of addresses, that are known to be from pesky bots. You should only do this if it is a REAL problem.

Code:
order allow,deny
deny from 192.168.
deny from 10.0.0.
allow from all
The above code denies access for a class 2 (192.168) and class 3 (10.0.0) address range. The later blocks 256 addresses, the former 256*256 addresses.

I usually add this when I get hit by scrapers. They can come in and hit you hard. You really can't go by their "user-agent" element, for it changes. Better to nip the IP address in the bud.
Dodger is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
i think important question ? Google Link operator related question thenetspiders Google Web Search 3 10-28-2010 05:07 AM
Quick question about drop down menus and invisible text Stemage Search Engine Optimization 3 02-06-2006 12:02 PM
The Million Dollar Question Melwood Search Engine Optimization 4 11-22-2005 07:37 AM
Weekly SEO Quiz Question dyn4mik3 The Padded Room 5 06-17-2005 10:30 PM
Question regarding spidering multiple parameters DamienClothier Dynamic Website and Technical Issues 0 07-27-2004 11:58 AM