Results 1 to 2 of 2

Thread: robot text

  1. #1
    Join Date
    Nov 2005

    Default robot text

    Ok, now I have been reading all night about site maps, now I have come across Robot Text, headache coming on...... is this important to have in your index page too? I have generated the text as explained on another thread, where do I then put the text and how do I link it to my index page? So many questions, so little time.... ha ha.

    Thanks in advance to anybody that may reply to me.


  2. #2
    Join Date
    Sep 2005

    Default Re: robot text

    It's not necessary to have a robots.txt file unless there are pages you don't want a bot to spider, or if you want to ban a particular bot. If you want to add a blank robots.txt file for peace of mind, simply save a blank notepad page as 'robots.txt' and upload it to your root directory.

    I have a robots.txt file which contains the following:

    User-agent: Rufusbot
    Disallow: /

    What this means is that Rufusbot is banned from spidering my site. If you want to ban all bots (you don't) your file would read:

    User-agent: *
    Disallow: /

    Or if you wanted to stop a bot crawling a particular area:

    User-agent: googlebot
    Disallow: /cgi-bin/

    (replace cgi bin with the page or area you don't want indexed).

    Hope this is of some use.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts