+ Reply to Thread
Results 1 to 11 of 11

Thread: blocking robots
      
   

  1. #1
    mattyfergus is offline Second Lieutenant
    Join Date
    Sep 2008
    Posts
    121

    Default blocking robots

    How do i block robots from crawling certain pages on my site?

    Thanks in advance

  2. #2
    Join Date
    Feb 2006
    Location
    Earth
    Posts
    8,719

    Default Re: blocking robots

    Hi, depends on what you want the robot to do. IE: Just not index the page, but still follow the links on the page? Or not index anything.
    Place one of these in the head of each of the pages to not index.



    <META name="robots" content="noindex">

    0r

    <META name="robots" content="noindex,nofollow">


  3. #3
    mattyfergus is offline Second Lieutenant
    Join Date
    Sep 2008
    Posts
    121

    Default Re: blocking robots

    Thanks for the reply.

    When you say the HEAD OF THE PAGE, where exactly do you mean?

  4. #4
    mattyfergus is offline Second Lieutenant
    Join Date
    Sep 2008
    Posts
    121

    Default Re: blocking robots

    Also, I dont want the robots to see the text as it is duplicate text and google says this will degrade the site. So what code should I use?

    Thanks again.

  5. #5
    mattyfergus is offline Second Lieutenant
    Join Date
    Sep 2008
    Posts
    121

    Default Re: blocking robots

    Anybody?

  6. #6
    Join Date
    Feb 2006
    Location
    Earth
    Posts
    8,719

    Default Re: blocking robots

    <META name="robots" content="noindex,nofollow">

    The code above should do it. place it in the head of all page,s via. Main menu>View>Page HTML>Between head tag, paste the code in, & press ok.

  7. #7
    Karen Mac's Avatar
    Karen Mac is offline General
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,354

    Default Re: blocking robots

    Rather you index those pages or not, or no follow, you can still be penalized for duplicate content if it exists. No index means the robot wont spider and add it to cache, doesnt mean they wont rank it.

    No Follow doesnt mean they wont INDEX.. they will, it just means you dont VOUCH for the content of the link..

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    07031847328 / United Kingdom

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  8. #8
    Join Date
    Feb 2006
    Location
    Earth
    Posts
    8,719

    Default Re: blocking robots

    Quote Originally Posted by Karen Mac View Post
    Rather you index those pages or not, or no follow, you can still be penalized for duplicate content if it exists. No index means the robot wont spider and add it to cache, doesnt mean they wont rank it.

    Karen
    So would you say to stop the robots on the page that links are coming from to that page that is not to be indexed, or add a tag like this <meta name="robots" content="none"> so its not looked at by the robots??

  9. #9
    Karen Mac's Avatar
    Karen Mac is offline General
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,354

    Default Re: blocking robots

    To stop the robots yes... its NO INDEX.... or deny them access in htaccess file, however, duplicate content is duplicate content.. and we all know GOOGLE will look and find it.

    If you want to remove a url in cache or index, or search results.. you can contact google to remove the url from their index... still doesnt mean they dont have a record of it

    NO FOLLOW doesnt mean they wont index.. they will.. it means you dont vouch or vote for the content where the link leads


    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    07031847328 / United Kingdom

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  10. #10
    Join Date
    Feb 2006
    Location
    Earth
    Posts
    8,719

    Default Re: blocking robots

    Thanks Karen, good one to keep in mind.

    Mattyfurgus, we have been told. Ya just cant hide from the little devils. LOL

  11. #11
    mattyfergus is offline Second Lieutenant
    Join Date
    Sep 2008
    Posts
    121

    Default Re: blocking robots

    Thanks people.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

     

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49