Results 1 to 15 of 15

Thread: Robot.txt
      
   

  1. #1
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Robot.txt

    Sorry all, I'm back with another question.

    I want google to spider certain pages (metatags, images, ect.) I have but not post the links to them in a search.

    Do I use the Robot.txt for this, or does that block the spiders from looking at the page at all.

    If not is there a way to do this?

    Thanks in advance.

  2. #2
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,332

    Default Re: Robot.txt

    No youd use a sitemap for this. Generally you use robots.txt files to tell it what NOT to spider.

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  3. #3
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    Ohhhhhhhhhh.......... Thanks........ But...........

    how do you tell the SE's which pages not to show in a sitemap?

    :)

  4. #4
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,332

    Default Re: Robot.txt

    Dont put them in the sitemap!

    Then use the robots file to disallow pages you dont want indexed.

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  5. #5
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    Oh so if I don't put them in the sitemap
    the spiders will scan the pages but won't put the link in searches.
    Thanks again you've been a big help. :)

  6. #6
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,332

    Default Re: Robot.txt

    Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  7. #7
    Bethers's Avatar
    Bethers is offline Major General & Forum Moderator
    Join Date
    Feb 2006
    Posts
    5,224

    Default Re: Robot.txt

    What good does it do to have them spider the page if you don't want them to give the links to it? This doesn't make any sense. If you don't want them to give the links, they you shouldn't have them spider it.

    I think you need to explain this better - cuz you're talking *****s and oranges.

  8. #8
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    I'm sorry for being so obtuse.. this is all just very new to me.

    I understand that the robot.txt file blocks the spiders from seeing the page alltogether..

    and that the site map tells them what to search and index in searches...

    my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...

    but not put the url of that specific page in to their searches
    so potental customers won't come in for the first time to the wrong page and leave the site without seeing what we really carry.

  9. #9
    Bethers's Avatar
    Bethers is offline Major General & Forum Moderator
    Join Date
    Feb 2006
    Posts
    5,224

    Default Re: Robot.txt

    You can't have it both ways - and EVERY page to your site should be being optimized as an entry page - and for different keywords - so that you'll be able to have so many different pages for people to enter. If a customer wants to find blue widgets - they are much better off entering on the exact page that sells them than having to search your site to find them - so optimize the pages correctly - and let the se's send people to them directly. Don't fight what is good for you.

  10. #10
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    the main reason is that I have pages that only hold data for other pages

  11. #11
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    I guess I'll just leave them then thanks for all of your help...

    I know I didn't make it easy for you... sorry and thanks again

  12. #12
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,332

    Default Re: Robot.txt

    my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...
    Thats what you use alt tags for. They dont SEE images, they can read the file names and the alt tags tho.

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



  13. #13
    Join Date
    Jun 2007
    Location
    In front of my computer.
    Posts
    442

    Default Re: Robot.txt

    Thanks again karen. I think I got it now

  14. #14

    Default Re: Robot.txt

    Quote Originally Posted by Karen Mac View Post
    Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

    Karen
    Inquiring minds want to know............
    What happens if someone has BOTH sitemap and robot.text?
    Where's Your Life Going? You might find some help here: http://www.ourjourneytolife.com/banners3.html
    Who is Pristinelife? http://ourjourneytolife.com/aboutme.html



  15. #15
    Join Date
    Apr 2006
    Location
    X marks the spot
    Posts
    8,332

    Default Re: Robot.txt

    One would get gas which would implode internally and give one mad cow ******* if one could only read!

    Originally Posted by Karen Mac
    Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file
    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)



Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

     

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •