Thread: blocking robots

How do i block robots from crawling certain pages on my site?

Thanks in advance

Hi, depends on what you want the robot to do. IE: Just not index the page, but still follow the links on the page? Or not index anything.
Place one of these in the head of each of the pages to not index.



<META name="robots" content="noindex">

0r

<META name="robots" content="noindex,nofollow">

Thanks for the reply.

When you say the HEAD OF THE PAGE, where exactly do you mean?

Also, I dont want the robots to see the text as it is duplicate text and google says this will degrade the site. So what code should I use?

Thanks again.

Anybody?

<META name="robots" content="noindex,nofollow">

The code above should do it. place it in the head of all page,s via. Main menu>View>Page HTML>Between head tag, paste the code in, & press ok.

Rather you index those pages or not, or no follow, you can still be penalized for duplicate content if it exists. No index means the robot wont spider and add it to cache, doesnt mean they wont rank it.

No Follow doesnt mean they wont INDEX.. they will, it just means you dont VOUCH for the content of the link..

Karen

Originally Posted by Karen Mac

Rather you index those pages or not, or no follow, you can still be penalized for duplicate content if it exists. No index means the robot wont spider and add it to cache, doesnt mean they wont rank it.

Karen

So would you say to stop the robots on the page that links are coming from to that page that is not to be indexed, or add a tag like this <meta name="robots" content="none"> so its not looked at by the robots??

To stop the robots yes... its NO INDEX.... or deny them access in htaccess file, however, duplicate content is duplicate content.. and we all know GOOGLE will look and find it.

If you want to remove a url in cache or index, or search results.. you can contact google to remove the url from their index... still doesnt mean they dont have a record of it

NO FOLLOW doesnt mean they wont index.. they will.. it means you dont vouch or vote for the content where the link leads


Karen

Thanks Karen, good one to keep in mind.

Mattyfurgus, we have been told. Ya just cant hide from the little devils. LOL

Thanks people.