Robots file

mfbisson · #1 01-22-2009, 01:49 AM

Hi, can anyone tell me how to put a robots.txt file on my site? I have been reading about but i cant figure it out. Why doesn't BV have a spot to generate one. Should I even bother.
www.foolsdiet.com

Marincky · #2 01-22-2009, 01:51 AM

What do you require the robots file for? If you are happy for the SE's to visit all the pages on your site then don't bother. If you are trying to restrict them let us know? You were very vague with your question

Vasili · #3 01-22-2009, 02:00 AM

Quote:

Originally Posted by mfbisson

Hi, can anyone tell me how to put a robots.txt file on my site? I have been reading about but i cant figure it out. Why doesn't BV have a spot to generate one. Should I even bother.
www.foolsdiet.com

It's not really important anymore, unless you wish the bots ignore/not follow specific pages (for whatever reason), and only then would you be advised to create a robots.txt file with the appropriate "do not follow" instructions.

Even though Google has significantly devalued a lot of the elemental metric values we typically consider in "optimizing" ourt websites, it is still advised to create and install a proper XML file for your website once it is completed. There are a lot of threads with more advice on how to do this in these forums, so just keep it in mind as one of the "last things to do" when designing your website!

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)