Unheard of but I need advice

[Jazajay]

I know it's unheard of but I need some geeky techy advice. The world has finally come to an end I know :(

I've recently discovered the new header spec.

X-Robots-Tag: NOINDEX

I've tried every way I can think of to implant this header but every time I check my headers it doesn't appear to be there. Obviously broken headers can skrew you so I'm hesitant about moving this from my testing server up to my deveolpment server until I'm positive it's correct.

What I've tried
header("X-Robots-Tag: NOINDEX")

<file /cat.php>
set .... X-Robots-Tag: NOINDEX
</files>

In the .htaccess. I know thats not how you write that directive but I have implanted it properly I insure you.
I've tried adding a .htaccess file with that directive to that sub dir but still no joy.

Does any one have any more suggestions?
Any help would be greatly appreicated.
Thanks in advance and thanks for reading
Jaza

[AussieWebmaster]

This may help
http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html
If not can find something simpler

[Jazajay]

Ha
That was my second resource after an email I received.
But cheers for the link, if any one doesn't know the spec definatley check that link out, worth a read.

I'm still having issues with the implantation even after I view the http spec as the link you provide suggests.

Any ideas on implantation AussieWebmaster?

Have you used the tag and if so how?

[AussieWebmaster]

I am not using the tag... but have been considering rolling it out to a couple of sites to see how it impacts

[Jazajay]

Yeah I liked the spec, geek I know, in fact it made my day, my miss' didn't see the appeal but hay everyone's a critique. Anyway I digress.

I've implanted it every way I can think of but when I check the HTTP headers sent back it's not there?

Now surly it's meant to be sent right? I mean it's a HTTP header after all. If it's not sent back G's just released a spammers dream tool as you cant track it.

What do you think?
Has Matt said something I've missed about tracking it or is there something up in my implantation?

I'm going to go with my implantation. I could be wrong, unheard of but I suppose I could be :)

[cscgal]

How does this differ from using a robots.txt file? I understand that with a dynamic site you can generate on-the-fly whether you want the particular page to contain the directive. However, a robots.txt file can be dynamically generated as well.

[AussieWebmaster]

How does this differ from using a robots.txt file? I understand that with a dynamic site you can generate on-the-fly whether you want the particular page to contain the directive. However, a robots.txt file can be dynamically generated as well.

But sometimes the robots.txt file is not stopping the spiders

[cscgal]

If that's the case, I can't see this directive working any better. It seems to just be an alternative way of overriding a robots.txt file on a page by page basis.

If a spider isn't listening to a robots.txt file, are they really listening to this?

[Jazajay]

I relized my problem after talking with Beu. I misread the email I recieved about it.
This directive is only avaliable to non html documents - pdf's etc...
Hence my problem.

[AussieWebmaster]

If that's the case, I can't see this directive working any better. It seems to just be an alternative way of overriding a robots.txt file on a page by page basis.

If a spider isn't listening to a robots.txt file, are they really listening to this?

true... there is only best practices which you have - the spiders that seem to ignore them are going to do it reagrdless...

[cscgal]

See, I'm always right :)

[Jazajay]

Yeah i know cscgal same here :D

[Bhumika]

well said cscgal :) you are always right ha