Announcement

Collapse
No announcement yet.

robot text

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • robot text

    Ok, now I have been reading all night about site maps, now I have come across Robot Text, headache coming on...... is this important to have in your index page too? I have generated the text as explained on another thread, where do I then put the text and how do I link it to my index page? So many questions, so little time.... ha ha.

    Thanks in advance to anybody that may reply to me.

    Jan

  • #2
    Re: robot text

    It's not necessary to have a robots.txt file unless there are pages you don't want a bot to spider, or if you want to ban a particular bot. If you want to add a blank robots.txt file for peace of mind, simply save a blank notepad page as 'robots.txt' and upload it to your root directory.

    I have a robots.txt file which contains the following:

    User-agent: Rufusbot
    Disallow: /

    What this means is that Rufusbot is banned from spidering my site. If you want to ban all bots (you don't) your file would read:

    User-agent: *
    Disallow: /

    Or if you wanted to stop a bot crawling a particular area:

    User-agent: googlebot
    Disallow: /cgi-bin/

    (replace cgi bin with the page or area you don't want indexed).

    Hope this is of some use.

    Comment

    Working...
    X