mphung
12-12-2006, 04:35 PM
I'm not well-versed in robots.txt so I'm hoping someone with more expertise can answer what might be a simple question.
I'm want to disallow crawling of pages with dynamic variables. Currently one of our very large sites which appends a lot of variables to the end of nearly all our pages.
Example 1: domain.com/help/?variable1=x&variable2=y&variable3=&zipcode=&
where everything after and including the "?" is not necessary to view the page (i.e., duplicate). The file domain.com/help/index.asp doesn't exist. The version of the page that needs to be indexed, and the only one that should be indexed, is domain.com/help/
So, will Disallow: /help/ still make the domain.com/help/ page itself available?
Example 2: domain.com/content.asp?pageid=123&variable1=&...
where pageid=123 is unique to the page, but everything after isn't. So I want domain.com/content.asp?pageid=123, ?pageid=234, ?pageid=345 to be indexed, but not anything beyond that. Is there a way to include the first variable+value but disallow subsequent variable strings with robots.txt?
Any and all input is greatly appreciated.
Thanks.
I'm want to disallow crawling of pages with dynamic variables. Currently one of our very large sites which appends a lot of variables to the end of nearly all our pages.
Example 1: domain.com/help/?variable1=x&variable2=y&variable3=&zipcode=&
where everything after and including the "?" is not necessary to view the page (i.e., duplicate). The file domain.com/help/index.asp doesn't exist. The version of the page that needs to be indexed, and the only one that should be indexed, is domain.com/help/
So, will Disallow: /help/ still make the domain.com/help/ page itself available?
Example 2: domain.com/content.asp?pageid=123&variable1=&...
where pageid=123 is unique to the page, but everything after isn't. So I want domain.com/content.asp?pageid=123, ?pageid=234, ?pageid=345 to be indexed, but not anything beyond that. Is there a way to include the first variable+value but disallow subsequent variable strings with robots.txt?
Any and all input is greatly appreciated.
Thanks.