Announcement

Collapse
No announcement yet.

Robot.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Robot.txt

    Sorry all, I'm back with another question.

    I want google to spider certain pages (metatags, images, ect.) I have but not post the links to them in a search.

    Do I use the Robot.txt for this, or does that block the spiders from looking at the page at all.

    If not is there a way to do this?

    Thanks in advance.
    http://www.penniesnpence.com

  • #2
    Re: Robot.txt

    No youd use a sitemap for this. Generally you use robots.txt files to tell it what NOT to spider.

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


    Comment


    • #3
      Re: Robot.txt

      Ohhhhhhhhhh.......... Thanks........ But...........

      how do you tell the SE's which pages not to show in a sitemap?

      :)
      http://www.penniesnpence.com

      Comment


      • #4
        Re: Robot.txt

        Dont put them in the sitemap!

        Then use the robots file to disallow pages you dont want indexed.

        Karen

        VodaHost

        Your Website People!
        1-302-283-3777 North America / International
        02036089024 / United Kingdom
        291916438 / Australia

        ------------------------

        Top 3 Best Sellers

        Web Hosting - Unlimited disk space & bandwidth.

        Reseller Hosting - Start your own web hosting business.

        Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


        Comment


        • #5
          Re: Robot.txt

          Oh so if I don't put them in the sitemap
          the spiders will scan the pages but won't put the link in searches.
          Thanks again you've been a big help. :)
          http://www.penniesnpence.com

          Comment


          • #6
            Re: Robot.txt

            Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

            Karen

            VodaHost

            Your Website People!
            1-302-283-3777 North America / International
            02036089024 / United Kingdom
            291916438 / Australia

            ------------------------

            Top 3 Best Sellers

            Web Hosting - Unlimited disk space & bandwidth.

            Reseller Hosting - Start your own web hosting business.

            Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


            Comment


            • #7
              Re: Robot.txt

              What good does it do to have them spider the page if you don't want them to give the links to it? This doesn't make any sense. If you don't want them to give the links, they you shouldn't have them spider it.

              I think you need to explain this better - cuz you're talking *****s and oranges.
              Beth
              A Child's Palace - Pinata Palace - Moxie Enterprises

              SEO and Marketing Tools
              SEO - The Basics

              Comment


              • #8
                Re: Robot.txt

                I'm sorry for being so obtuse.. this is all just very new to me.

                I understand that the robot.txt file blocks the spiders from seeing the page alltogether..

                and that the site map tells them what to search and index in searches...

                my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...

                but not put the url of that specific page in to their searches
                so potental customers won't come in for the first time to the wrong page and leave the site without seeing what we really carry.
                http://www.penniesnpence.com

                Comment


                • #9
                  Re: Robot.txt

                  You can't have it both ways - and EVERY page to your site should be being optimized as an entry page - and for different keywords - so that you'll be able to have so many different pages for people to enter. If a customer wants to find blue widgets - they are much better off entering on the exact page that sells them than having to search your site to find them - so optimize the pages correctly - and let the se's send people to them directly. Don't fight what is good for you.
                  Beth
                  A Child's Palace - Pinata Palace - Moxie Enterprises

                  SEO and Marketing Tools
                  SEO - The Basics

                  Comment


                  • #10
                    Re: Robot.txt

                    the main reason is that I have pages that only hold data for other pages
                    http://www.penniesnpence.com

                    Comment


                    • #11
                      Re: Robot.txt

                      I guess I'll just leave them then thanks for all of your help...

                      I know I didn't make it easy for you... sorry and thanks again
                      http://www.penniesnpence.com

                      Comment


                      • #12
                        Re: Robot.txt

                        my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...
                        Thats what you use alt tags for. They dont SEE images, they can read the file names and the alt tags tho.

                        Karen

                        VodaHost

                        Your Website People!
                        1-302-283-3777 North America / International
                        02036089024 / United Kingdom
                        291916438 / Australia

                        ------------------------

                        Top 3 Best Sellers

                        Web Hosting - Unlimited disk space & bandwidth.

                        Reseller Hosting - Start your own web hosting business.

                        Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


                        Comment


                        • #13
                          Re: Robot.txt

                          Thanks again karen. I think I got it now
                          http://www.penniesnpence.com

                          Comment


                          • #14
                            Re: Robot.txt

                            Originally posted by Karen Mac View Post
                            Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

                            Karen
                            Inquiring minds want to know............
                            What happens if someone has BOTH sitemap and robot.text?
                            Where's Your Life Going? You might find some help here: http://www.ourjourneytolife.com/banners3.html
                            Who is Pristinelife? http://ourjourneytolife.com/aboutme.html


                            Comment


                            • #15
                              Re: Robot.txt

                              One would get gas which would implode internally and give one mad cow ******* if one could only read!

                              Originally Posted by Karen Mac
                              Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file
                              Karen

                              VodaHost

                              Your Website People!
                              1-302-283-3777 North America / International
                              02036089024 / United Kingdom
                              291916438 / Australia

                              ------------------------

                              Top 3 Best Sellers

                              Web Hosting - Unlimited disk space & bandwidth.

                              Reseller Hosting - Start your own web hosting business.

                              Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


                              Comment

                              Working...
                              X