PDA

View Full Version : do not spider script?



NightWolf
02-17-2007, 04:21 AM
Does anyone know a simple script to keep certain pages from being spidered? (i.e.- product download or members only pages.)

Bethers
02-17-2007, 05:48 AM
Nightwolf - you want to put that into the meta tags - using the robots.txt no index for those pages.

Here's some directions on it:

http://www.robotstxt.org/wc/meta-user.html
and more:
http://www.outfront.net/tutorials_02/adv_tech/robots.htm