PDA

View Full Version : Robot.txt



PennieNPence
08-27-2007, 08:22 PM
Sorry all, I'm back with another question.

I want google to spider certain pages (metatags, images, ect.) I have but not post the links to them in a search.

Do I use the Robot.txt for this, or does that block the spiders from looking at the page at all.

If not is there a way to do this?

Thanks in advance.

Karen Mac
08-28-2007, 04:26 AM
No youd use a sitemap for this. Generally you use robots.txt files to tell it what NOT to spider.

Karen

PennieNPence
08-28-2007, 04:38 AM
Ohhhhhhhhhh.......... Thanks........ But...........

how do you tell the SE's which pages not to show in a sitemap?

:)

Karen Mac
08-28-2007, 05:58 AM
Dont put them in the sitemap!

Then use the robots file to disallow pages you dont want indexed.

Karen

PennieNPence
08-28-2007, 05:26 PM
Oh so if I don't put them in the sitemap
the spiders will scan the pages but won't put the link in searches.
Thanks again you've been a big help. :)

Karen Mac
08-30-2007, 04:10 AM
Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

Karen

Bethers
08-30-2007, 04:46 AM
What good does it do to have them spider the page if you don't want them to give the links to it? This doesn't make any sense. If you don't want them to give the links, they you shouldn't have them spider it.

I think you need to explain this better - cuz you're talking *****s and oranges.

PennieNPence
08-30-2007, 04:49 AM
I'm sorry for being so obtuse.. this is all just very new to me.

I understand that the robot.txt file blocks the spiders from seeing the page alltogether..

and that the site map tells them what to search and index in searches...

my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...

but not put the url of that specific page in to their searches
so potental customers won't come in for the first time to the wrong page and leave the site without seeing what we really carry.

Bethers
08-30-2007, 05:13 AM
You can't have it both ways - and EVERY page to your site should be being optimized as an entry page - and for different keywords - so that you'll be able to have so many different pages for people to enter. If a customer wants to find blue widgets - they are much better off entering on the exact page that sells them than having to search your site to find them - so optimize the pages correctly - and let the se's send people to them directly. Don't fight what is good for you.

PennieNPence
08-30-2007, 05:23 AM
the main reason is that I have pages that only hold data for other pages

PennieNPence
08-30-2007, 05:25 AM
I guess I'll just leave them then thanks for all of your help...

I know I didn't make it easy for you... sorry and thanks again

Karen Mac
09-01-2007, 05:24 PM
my problem is that I wanted the spiders to see the elements (buttons,images,ect.) on a page...



Thats what you use alt tags for. They dont SEE images, they can read the file names and the alt tags tho.

Karen

PennieNPence
09-01-2007, 07:05 PM
Thanks again karen. I think I got it now

pristinelife
09-04-2007, 06:08 PM
Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file

Karen

Inquiring minds want to know............
What happens if someone has BOTH sitemap and robot.text?

Karen Mac
09-04-2007, 06:16 PM
One would get gas which would implode internally and give one mad cow ******* if one could only read!



Originally Posted by Karen Mac http://www.vodahost.com/vodatalk/images/buttons/viewpost.gif (http://www.vodahost.com/vodatalk/google/26269-robot-txt.html#post178813)
Oy Vey.. no you put links in a sitemap you do want spidered. If you use one of the generators you can go edit out the pages you dont want spidered, or in some cases you can tell it which pages to omit at the start. So.. If you dont have a sitemap and you dont have links to the pages, how do u want them found?... You do a combo of the site map and also do a robots.txt file


Karen