Robots.txt Generator

[mcanerin]

Hi All,

As a Mod here I try not to link to my own stuff too much, but I thought this would be useful.

I constantly get clients asking me what a robots.txt file is, how to use it, etc. Especially after I tell them they don't have one and should get one.

So I just made a (free) robots.txt generator and info page - hopefully to answer these questions and to make my (and other people's) life easier.

http://www.mcanerin.com/search-engine/robots-txt.htm

Hopefully it's useful to someone here - I know I've needed it a few times personally.

In the spirit of this section of the forum, PLEASE notify me (or post a reply) if you find a bug, bad info, suggested improvements etc.

Cheers,

Ian

[Nacho]

Hey Ian, you're tool rocks! Glad you posted it.

Here are a couple great posts here at SEW for robot.txt discussion:

Getting millions of dynamic pages indexed (in my favorite subscirptions)
Let's discuss ROBOTS.TXT

Saludos!

[mcanerin]

Arrrg!

I just tested it more thoroughly in Firefox and discovered that the javascript seems to spit out "Same as default" rather than treating it properly as nothing.

Works fine in IE. Any coders out there that could suggest a workaround that works for both? I've tried several things and so far no luck - though I admit I'm not much of a programmer.

damn...

Ian

[NetinsertGuy]

Very useful resource for understanding and defining robots.txt. The tool is easy to use and works fine. Thanks a lot for sharing!

[orion]

This is a handy tool with excellent information. I would suggest the following

Positioning

1. Call the script from an external .js file
2. Place the script at the beginning of the page, followed by the information supplied.


Programming

The JavaScript need to be optimized. Thus the file size is reduced and coding is improved. Try this,

1. In the function create(form), precache repetitive DOM "dot" calls

eg.

var x =form.robolist;

then use x.value where needed.

Each time you save a dot call is less interaction with the browser and JavaScirpt dom. Less interactions means improvement in processing speed. This is also in line with good programming practices.

2. Use a for-to loop to get rid of all the "If" conditional statements. Here you can use the eval command or an array, either one with a for-to loop; i.e.,

for(var i=0;i<n;i++)
{
'
'
'
}

This should reduce the file instructions and size.

Finally, the external file can be locked into the domain and javascript-encrypted.

The pull-down menus can be replaced by mutually exclusive radio buttons, but this is a matter of personal taste and not a biggie.


Overall, is an excellent tool.


Orion

[mcanerin]

Well, I incorporated most of Orions suggestions and also fixed it to work in Firefox as well as IE

Thanks, everyone!

Ian