Jazajay
02-24-2008, 05:36 PM
Hi guys and gals heres the snippet from live
on their crawlers -
....our crawler will include the "If-None-Match" header and the ETag value in the GET request.
What they are saying is they will only crawl your page if it hasn't been modified.
But what if the etag is removed? What happens with the crawler?
Any ideas?
Jaza
on their crawlers -
....our crawler will include the "If-None-Match" header and the ETag value in the GET request.
What they are saying is they will only crawl your page if it hasn't been modified.
But what if the etag is removed? What happens with the crawler?
Any ideas?
Jaza