Thread: AWstats

hi can anyone help me with 404 error reports in awstats for my site.
I'm only really concerned with the error hits i'm getting from robots

this is what it reads

Url
/robots.txt
error hits
38

Not really sure how to rectify this, hope one of the pro's can explain how.

many thanks

Max

Do you have a 'robots.txt' file? I believe this is basically used to tell bots which pages NOT to read/cache. The bot looks for it and if it doesn't find robots.txt it knows it can read any/all the pages.
BTW: I am far from a pro! Perhaps if I am incorrect, someone will enlighten both of us.

jcaywood pretty much hit the nail on the head. Search engines are trying to access the file called "robots.txt", which since you don't have one its not being found, which causes a 404 error (page not found). If you create a robots.txt, even a blank one, you won't get a 404 error when trying to access it.

Thanks for your help guys, I will work on it.

much appreciated