I'm new to all of this, but have a question. When I did an advanced search for our site, with the domain name included, all of the links that were shown were from our old site version. This would explain why we weren't ranked when running Web Position Gold, but how can I get the new pages indexed, or do I just have to wait for the MSNbot to come around? It's already been at least 3 months.

Also, how do I find out when the bots have visited our site? We are using Clicktracks to analyze traffic, but I haven't found a way to distinguish between the difference bots. Do I have to manually look through each logfile from each day searching either for "Googlebot" or a specific IP address?

Thanks for any help that you can provide me with!! I've been searching and really haven't been able to come up with any concrete answers.

The last I checked, MSN is still using the Inktomi index as it is still readying its own search engine. They have beta tested it a couple of times now but your results probably came from what is in the Inktomi index. Therefore, I'd simply recommend that you wait it out until MSN finally implements their own crawler based results.

Thank you for your response.

Do you know how to check log files to see if they have been spidered? I know this may be a silly, basic question, but as I said, I'm new to all this, and appreciate all the information I can get!

To the best of my knowledge, Click Tracks doesn't break down reports on spider visits. Pretty sure that NetTracker does. There's a few programs out there that will parse this info for you... not many for windows. If you're handy like that, I think it's not too difficult to build your own. Or, as you mentioned, you can manually review the files in a spreadsheet, providing they're not too large. If you find something good, I'd be interested to know.

Lex,
You are right about Clicktracks. I briefly spoke with them and they said they filter out spider visits so the only way to track them would be to individually search for the specific IP addresses, so this way would be quicker than going through each day of log files and each IP address...though I have a feeling there may even be too many to do that for! (?)

Thanks :)