Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engine Marketing Strategies > Search Engine Optimization
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 11-09-2005   #1
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Prevent Indexing While Testing New Site

Once I move publish my site to my web host provider, I will be testing my customized shopping cart. I would like to prevent any search engine indexing while I am testing. This time interval may be 2 weeks - difficult to estimate right now.

Some approaches I have considered are:
1. Setup my site w/ the host provider as an ip address and after testing change to my domain name.
2. Add a robots.txt file to my site root folder (I read on the google site that this file is recoginzed and google won't index the site. Not sure if other SEs will honor this file?)
3. Consideration given to having my host provider setting up a password on the default home page file (index.htm, default, etc.)

Are any one of these approaches alone sufficient to prevent indexing?

Also, if I prevent indexing of my site while testing, will this prevent or negatively affect future indexing of my site?

Right now the immediate need is to test without SE indexing. Is this possible and is there a downside to preventing indexing while testing?

Thanks much,
jl015
jl015 is offline   Reply With Quote
Old 11-09-2005   #2
Rynert
Member
 
Join Date: Oct 2005
Posts: 17
Rynert is on a distinguished road
Unless you actually want to avoid the SEs even knowing about your domain I would simply put up a default home page with either some company info or a 'coming soon' notice.

Then run your site from www.domain.com/real-homepage.asp

Only you know this URL, knowhere else links to it so nobody can find it.*

When you are done simply set real-homepage.asp to be the domain default, or rename to index.asp or whatever.

* I am assuming you have not already used the domain and are re-using existing page names.
Rynert is offline   Reply With Quote
Old 11-09-2005   #3
DarkMatter
Master Blaster
 
Join Date: Feb 2005
Location: New Jersey,USA
Posts: 137
DarkMatter is on a distinguished road
if the site is not linked from any previously indexed pages then you most likely would not get indexed, however i would not leave it to chance.

a robots.txt file is all you need. pages that aren't indexed can't be penalized so there are no drawbacks.
DarkMatter is offline   Reply With Quote
Old 11-09-2005   #4
Rynert
Member
 
Join Date: Oct 2005
Posts: 17
Rynert is on a distinguished road
I have had domains indexed by Google that have not been linked anywhere (whois?) so definately don't take the chance.
Rynert is offline   Reply With Quote
Old 11-09-2005   #5
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
I have the index page of a site sitting in the index now - title, description and all - that got crawled within a couple of hours of the domain resolving to the host, and was included in the index by the very next morning. Not one single, solitary link, even to this day there are none - and it had just been put on hosting.
Marcia is offline   Reply With Quote
Old 11-09-2005   #6
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
>>>Reply to Rynert<<<

Right now my domain is parked at godaddy with a coming soon page and I will be moving the domain to another provider for permanent hosting there. Correct, I have not used the domain yet - all of my development has been done on my local pc. I have also not populated any of my pages with <title> or any META TAGS for the search engine. Plan to add these after the cart testing.

I like your idea of the coming soon page with a different name for the default page and once testing complete rename to the permanent default page.


>>>Reply to DarkMatter<<<

Definitely going to use the robots.txt file. Do you happen to know if all SEs recognize the robots.txt file?


>>>General Comments<<<

Btw, because the domain has been parked at godaddy for several months, it already knows of the domain name.

If it were not for the shopping cart testing, I would not be asking these questions. This is totally opposite of what I really want to achieve going forward - once testing is complete, I want it known by all SEs.



Marcia, Rynert, DarkMatter... thanks for your suggestions and experiences!
jl015 is offline   Reply With Quote
Old 11-09-2005   #7
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
robots.txt is honored by the majors, but there are plenty out there that don't.

Would it work to put up a regular temporary index page and then just password protect the rest of the site, or at least access to the cart? That way you'd get a headstart on the indexing and crawl schedule process, and when you're ready to turn the site loose it'll get indexed quicker - and rank at MSN within mere weeks.
Marcia is offline   Reply With Quote
Old 11-09-2005   #8
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Reply to Marcia:

Ok, so... right now the site is recognized by google (actually the only one I tried) as it is "parked" at godaddy with a godaddy version of a coming soon page .

So, I will be moving the domain to the my host provider. Would this move mean that the SEs would have to re-learn the domain?

Let me see if I understand your suggestion... the regular temporary index page would be www.domainname.com/index.htm where the htm contains "Coming Soon" verbage? When testing is complete, publish the permanent index.htm page?

This temp index page would be open and available to the internet and the remainder of the site would be password protected? Leaving the index.htm open during the entire testing process would eliminate any re-learning of the site by SEs?

So, I asked the host provider if they could password protect at the site root and in so doing prevent entry at any other folders, sub folders, files? I didn't get a definitive "yes".

I like this idea if it is as I understand it above. THe only question I am not clear on is how can the other files/folders be password protected? Or can they be password protected? thereby preventing any entry into other files.

Perhaps the password protection question is for another forum?

But all in all I appreciate the suggestion and will definitely consider it.
jl015 is offline   Reply With Quote
Old 11-10-2005   #9
Reflect
Member
 
Join Date: Aug 2005
Posts: 18
Reflect is on a distinguished road
IP only and password prrotect

I would definatley just use an IP provided by the host, just don't transfer the domain name to it until done.

I also would setup the cart in a sub directory and the have the host password protect that.

I am going through that right now except the site was already in the SERPs as it started as a content only. I put the cart files in a sub directory and then password protected that directory via my Cpanel.Take care,

Brian
Reflect is offline   Reply With Quote
Old 11-10-2005   #10
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Brian, thanks for the tips! I am planning to use and IP vs the domain name and lock down at a minimum the cart folder.
jl015 is offline   Reply With Quote
Old 11-11-2005   #11
gomer
Member
 
Join Date: Jul 2004
Posts: 13
gomer is on a distinguished road
You could also try a non-standard homepage name. For example:

www.yourdomain.com/test.html

If you turn of directory browsing, there is no way that a search engine could find that page, but you of course could test that page by hitting it in your broswer. That page could lead to all other pages on the site. When you are ready to go live, you make test.html, index.html.

Oh yeah, and perhaps don't use the Google toolbar when visiting your pages. I have heard that the toolbar can lead spiders to new content. Don't have expereince with that, just read about it.

I don't like the suggested method of banning robots using robots.txt. When you remove that page and are ready to go live, I am always fearful that the search engines may still go by what your old robots.txt used to say.

Last edited by gomer : 11-11-2005 at 04:49 PM.
gomer is offline   Reply With Quote
Old 11-11-2005   #12
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Quote:
publish my site
jl015, you've mentioned "publishing" your pages. Are you using Front Page and IIS hosting?

There's no reason why the host shouldn't have a definitive answer for you about password protection if they know what they're doing.

Quote:
once testing complete rename to the permanent default page.
And do a 301 redirect so both pages don't get caught with duplicate content. Easy as pie with Apache, if that's what you'll be on. You'll also have to do a 301 redirect from non-www to www anyway, so better find out about these things and how they're done before making a final hosting decision.

Last edited by Marcia : 11-11-2005 at 05:32 PM.
Marcia is offline   Reply With Quote
Old 11-13-2005   #13
PhilC
Member
 
Join Date: Oct 2004
Location: UK
Posts: 1,657
PhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud ofPhilC has much to be proud of
There's a small problem with putting up an index 'coming soon' page, and using a different pagename for the real site. It's good to make a site fully functional in development, and having the wrong default index page can be a nuisance. If it matters, then I prefer to develope the site on a different, unknown domain, or even a sub-domain, so that everything works perfectly in the development site, and nothing needs to be changed when it's transfered to the live site.

A big advantage is that significant future changes can be tried and tested in the development domain before transfering them to the live site.
PhilC is offline   Reply With Quote
Old 11-13-2005   #14
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Reply to Gomer:

I have wondered about using the robots.txt to prevent indexing during test and basically ruining any hope of getting reindexed properly once I go live with the site.
jl015 is offline   Reply With Quote
Old 11-13-2005   #15
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Reply to Marcia:

I am using Macromedia products (DW, Coldfusion, etc.) and will be ftping (or publishing) my site to an IIS server.

Good suggestion to make sure the host provider is clear what I want to accomplish with the password protection and that I get assurance they know what they are talking about.

Btw, can you elaborate on the 301 redirect? Is this something setup in the web server or in a web page? non-wwww vs www? Please bear with me as I have in the past only scratched the surface of SEO and the 301 is new to me.
jl015 is offline   Reply With Quote
Old 11-13-2005   #16
gomer
Member
 
Join Date: Jul 2004
Posts: 13
gomer is on a distinguished road
Quote:
I have wondered about using the robots.txt to prevent indexing during test and basically ruining any hope of getting reindexed properly once I go live with the site.
While the chances are small of something bad happening from temporarily banning the site using a robots.txt, I personally would not do it. Search engines have too many bugs in them these days (www vs non-www, page jacking etc) which makes me fearful that they can screw up something small like this.
gomer is offline   Reply With Quote
Old 11-13-2005   #17
jl015
Member
 
Join Date: Nov 2005
Posts: 17
jl015 is on a distinguished road
Reply to Phil:

Good point about renaming my index page. I do so understand where you are coming from. What a pain that would be to have to change every one of those instances of the home page. Although, thank goodness for Dreamweaver templates. Still it would be very labor intensive.
jl015 is offline   Reply With Quote
Old 11-13-2005   #18
Marcia
 
Marcia's Avatar
 
Join Date: Jun 2004
Location: Los Angeles, CA
Posts: 5,476
Marcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond reputeMarcia has a reputation beyond repute
Quote:
Btw, can you elaborate on the 301 redirect? Is this something setup in the web server or in a web page? non-wwww vs www? Please bear with me as I have in the past only scratched the surface of SEO and the 301 is new to me.
It's done server-side and is one of the most important features a website needs to have. It's all very easy on Apache servers - or in some cases there is plenty of idiot-proof information out there telling how whatever needs to be done server-side for sites is done - URL rewriting, redirects when pages move, etc.

Compare this

http://www.microsoft.com/technet/pro...383b1bb08.mspx

with this

http://www.webmasterwoman.com/design...-homepage.html

Not so with IIS hosting, where some things have to be done at server root level (meaning the host, not you has to implement) and unless your host has the necessary knowledge beforehand, you'll find yourself needing to hunt down and pay for software to accomplish what's built in with Apache.

Most basic, elementary test possible, and somethng that MUST be done:

Email the host and ask how the index page of a site is redirected from non-www - example.com to www.example.com - ask how you can do it and let us know what they reply.

Another thing - using CF, will you be using Session IDs?

Find that all out now, before putting a site live, or re-think what kind of hosting you'll be using would be my sugggestion.

===============================

Added:

Incidentally, this is how it's done on IIS:

IIS: Redirect a domain...

Last edited by Marcia : 11-13-2005 at 07:58 PM.
Marcia is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off