Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Member's Lounge > Beta Test
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 11-16-2004   #1
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
Robots.txt Generator

Hi All,

As a Mod here I try not to link to my own stuff too much, but I thought this would be useful.

I constantly get clients asking me what a robots.txt file is, how to use it, etc. Especially after I tell them they don't have one and should get one.

So I just made a (free) robots.txt generator and info page - hopefully to answer these questions and to make my (and other people's) life easier.

http://www.mcanerin.com/search-engine/robots-txt.htm

Hopefully it's useful to someone here - I know I've needed it a few times personally.

In the spirit of this section of the forum, PLEASE notify me (or post a reply) if you find a bug, bad info, suggested improvements etc.

Cheers,

Ian
__________________
International SEO
mcanerin is offline   Reply With Quote
Old 11-16-2004   #2
Nacho
 
Nacho's Avatar
 
Join Date: Jun 2004
Location: La Jolla, CA
Posts: 1,382
Nacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to beholdNacho is a splendid one to behold
Hey Ian, you're tool rocks! Glad you posted it.

Here are a couple great posts here at SEW for robot.txt discussion:

Getting millions of dynamic pages indexed (in my favorite subscirptions)
Let's discuss ROBOTS.TXT

Saludos!
Nacho is offline   Reply With Quote
Old 11-16-2004   #3
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
Arrrg!

I just tested it more thoroughly in Firefox and discovered that the javascript seems to spit out "Same as default" rather than treating it properly as nothing.

Works fine in IE. Any coders out there that could suggest a workaround that works for both? I've tried several things and so far no luck - though I admit I'm not much of a programmer.

damn...

Ian
__________________
International SEO
mcanerin is offline   Reply With Quote
Old 11-16-2004   #4
NetinsertGuy
Organize the web - www.Netinsert.com
 
Join Date: Jun 2004
Posts: 11
NetinsertGuy is on a distinguished road
Very useful resource for understanding and defining robots.txt. The tool is easy to use and works fine. Thanks a lot for sharing!
NetinsertGuy is offline   Reply With Quote
Old 11-16-2004   #5
orion
 
orion's Avatar
 
Join Date: Jun 2004
Posts: 1,044
orion is a splendid one to beholdorion is a splendid one to beholdorion is a splendid one to beholdorion is a splendid one to beholdorion is a splendid one to beholdorion is a splendid one to behold
Exclamation

This is a handy tool with excellent information. I would suggest the following

Positioning

1. Call the script from an external .js file
2. Place the script at the beginning of the page, followed by the information supplied.


Programming

The JavaScript need to be optimized. Thus the file size is reduced and coding is improved. Try this,

1. In the function create(form), precache repetitive DOM "dot" calls

eg.

var x =form.robolist;

then use x.value where needed.

Each time you save a dot call is less interaction with the browser and JavaScirpt dom. Less interactions means improvement in processing speed. This is also in line with good programming practices.

2. Use a for-to loop to get rid of all the "If" conditional statements. Here you can use the eval command or an array, either one with a for-to loop; i.e.,

for(var i=0;i<n;i++)
{
'
'
'
}

This should reduce the file instructions and size.

Finally, the external file can be locked into the domain and javascript-encrypted.

The pull-down menus can be replaced by mutually exclusive radio buttons, but this is a matter of personal taste and not a biggie.


Overall, is an excellent tool.


Orion

Last edited by orion : 11-16-2004 at 11:08 AM. Reason: adding some lines
orion is offline   Reply With Quote
Old 11-18-2004   #6
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
Well, I incorporated most of Orions suggestions and also fixed it to work in Firefox as well as IE

Thanks, everyone!

Ian
__________________
International SEO
mcanerin is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off