Does anyone know a simple script to keep certain pages from being spidered? (i.e.- product download or members only pages.)
Announcement
Collapse
No announcement yet.
do not spider script?
Collapse
X
-
Re: do not spider script?
Nightwolf - you want to put that into the meta tags - using the robots.txt no index for those pages.
Here's some directions on it:
http://www.robotstxt.org/wc/meta-user.html
and more:
http://www.outfront.net/tutorials_02...ech/robots.htm
Comment