Thread: robots

Can someone explain to me about robots......I noticed this is showing up in my stats.....not sure what they are?

robots are software programmed to search websites for information.

You will probable have seen googlbot, msn, yahoo etc and these are search engine bots that index sites so that other people can search for info on the web.

You can allow them access by doing nothing, or you can create a robots.txt file to keep them out (dosnt always work)

David has hit the nail on the head there but if you'd like a bit more info and some useful tools to do with robots, look here

so are they a good thing?

They are good if you don't want search engines to access secure or private files. You can set up a robot.txt file to basically list the pages that you don't want to be crawled. Most search engines are pretty good at respecting your robot.txt files, but Some may not. I have used them before to block certain pages that I have under construction. This way when my site gets crawled, the SE's dont see a page with a hundred broken links. Not sure if that may adversely affect my SE rankings, so I just don't let them access that page.

Yes, robots visiting your site are a good thing - they are the only way you can end up listed in a search for your keywords in any of the search engines.