Hanuman07
10-11-2007, 08:11 AM
I have something that starts to look like a problem. I run a small forum with something like 10-20 topics per day. We have about 5,000 topics in total. Google eat almost 80% of my bandwidth meaning about 6Gb per month.
I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem?
I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference.
What I would like to do is to ask Googlebot to visit only one time per day...
The site is www. thailandsforum.se
I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem?
I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference.
What I would like to do is to ask Googlebot to visit only one time per day...
The site is www. thailandsforum.se