Hardly crawled at all!

[capcol]

Hello everyone -

I am still relatively new to SEO, but over the past several months I have been trying hard to get up to speed on the subject. Since our site redesign and launch in January 2004, I've submitted the site to dozens of search directories, including multiple URL submissions to Google and Yahoo. When I say 'multiple', I mean about 1-2 times/month over the past 6 months. This site is a college website so it's already listed in many directories and SEs, but we still want to get crawled regularly as our content does change. There aren't any no-robots META tags on any of the pages and the robots.txt allows any spider to come along and crawl us, although I've blocked access to a few directories.

However, when I look at our web logs, I'm finding very little spider activity, even from the minor bots. Over the past 3 months, I've gotten maybe 1-2 spider visits per month, total. Yahoo has crawled us only five times this year and Googlebot hasn't crawled us at all. :eek: I'm very perplexed by this. Our site still needs more optimization, but I think it does meet at least the most fundamental SEO guidelines. I've been careful to avoid practices that might get us blacklisted by major SEs. But even if our site isn't perfectly optimized, wouldn't it get at least a partial crawl from the bots? From what I can tell, they're not even getting as far as the home page.

Can I please get some suggestions on where I can look for problem areas to troubleshoot? I'm really at my wit's end here and would appreciate your advice.

Thanks,

- JC

[Opie1Canopie]

JC, I'm sure the many experts here on the site will provide you with a more detailed answer, but I couldn't help responding to your plea!

Do you have good incoming links from other, popular sites? This is absolutely the best way to get your site crawled. Google (at present) prefers to follow a link from a reputable site to your site.

Also, if you don't already have one, a site map will help provide the spiders a guide to your site.

Hope this helps,
Opie1

[capcol]

JC, I'm sure the many experts here on the site will provide you with a more detailed answer, but I couldn't help responding to your plea!


Thanks, Opie! I hope you're right because at this point I'm really stuck on what to do next.


Do you have good incoming links from other, popular sites? This is absolutely the best way to get your site crawled. Google (at present) prefers to follow a link from a reputable site to your site.

Also, if you don't already have one, a site map will help provide the spiders a guide to your site.


Being a college, we do have a fair number of college guides and local directories that link to us. Our home page gets a page rank 6 right now. I also recently submitted to many of the free directories I saw listed in the Other Search Engines & Directories (http://forums.searchenginewatch.com/forumdisplay.php?f=15) forum. We do have a site map that's accessible from the home page.

Now here's another question... many of our main navigation links on our home page are coded in javascript. Could that interfere with the spiders' ability to crawl the site?

Thanks,

- JC

[I, Brian]

Now here's another question... many of our main navigation links on our home page are coded in javascript. Could that interfere with the spiders' ability to crawl the site?

You may have just cracked the magic nut. :)

Replace the navigation with a HTML+CSS form for easiest indexing of internal content - or else add a <noscript> tag beneath the nav with your links in. As JavaScript isn't as universally as accessible for humans and spiders as HTML+CSS, that's a definite direction I'd suggest looking into.

Of course, there may be other issues - one of the commonest problem is dynamic database-drive sites choking spiders with session IDs. Your site wouldn't happen to be dynamic as well, would it??

[Kal]

Since our site redesign and launch in January 2004, I've submitted the site to dozens of search directories, including multiple URL submissions to Google and Yahoo. When I say 'multiple', I mean about 1-2 times/month over the past 6 months.
Hi JC - you've received some excellent advice here already, but let me just add that you absolutely, positively don't need to resubmit your site to search engines. In fact, if your site is already linked well, you don't need to submit at all because the robots will crawl other links and find you. Once your pages are indexed, the robot will come back and crawl them on a regular basis according to their schedule so resubmitting does absolutely nothing but waste your time and theirs.

I don't believe resubmission attracts any penalties these days (except in extreme cases) but it is still a pointless exercise. Channel your time into locating high quality relevant sites related to yours and politely request links from them. If you can get them to link to you without having to reciprocate, that would be better for your link popularity.

[David Wallace]

Now here's another question... many of our main navigation links on our home page are coded in javascript. Could that interfere with the spiders' ability to crawl the site?
If that is the only way to reach any other pages on your site then yes that will stop spiders in their tracks. Add some text links in the footer and/or create a site map that then provides standard "href" links to your sub pages.