Thread: do not spider script?

Does anyone know a simple script to keep certain pages from being spidered? (i.e.- product download or members only pages.)

Nightwolf - you want to put that into the meta tags - using the robots.txt no index for those pages.

Here's some directions on it:

http://www.robotstxt.org/wc/meta-user.html
and more:
http://www.outfront.net/tutorials_02...ech/robots.htm