Thread: robots.txt

Now that my site is up, I'm looking to promote it and I ran up on robots.txt. I've looked at it and it helps or directs search engine spiders when they crawl your site. Now, my question is, does this have to be put in manually, or does BV put it in automaticly? If I do, next step is how and what to include?
Thanks all.
Vinny

One of the best ways to promote your site is to be sure you have good metatags. After you have those established, there are also many programs, such as exitexchange, which help you get more traffic.

I am not too familiar with robots.txt, but I have read somewhere that it can actually prevent robots from entering your website. I will have to look further into it.

I will get back to you about it.

Thanks a lot Steven, I happened accross this and was confused.

Hi Vinny

Here is an Interesting site I was looking at http://www.websight.net.au/search-en...r-friendly.asp

Thanks a lot it is a good site for reference.

I am learning every day about page rank ect... I will share with something with this forum I may be right or I may be wrong. I belong to many forums. this one forum and its members were pretty angry at this one site that was basically giving them a hard time. Anyway within a week there comments were posted on google and yahoo in the number two position right under the company they were angry with. The company ranked number 1 in there catagory the forums comments number 2. How did they get such a fast page rank? I think it was because of the robot txt inserted into their source code. In fact about 80% of pages that rank high seem to have robot txt. take a look. This is something i am still investingating. pet121

Thanks a lot, I'll take a look. The more I look into this the more questions a have. But there's something to it.
Vinny

Add this to your meta

<meta name="ROBOTS" content="INDEX,FOLLOW">

Thanks Dave, I'll do it today.

Originally Posted by Stroke.nz

Add this to your meta

<meta name="ROBOTS" content="INDEX,FOLLOW">

vinny make sure you place the above meta tag in each page

Originally Posted by Maddog

vinny make sure you place the above meta tag in each page

Thanks Maddog, wiil do!

Just thought of something - duh! I don't have any meta tags set up for my link page. Should I? It just links from my home page.

Another way that worked for me is to put as many links to popular sites on your website and do lots of link exchanges with popular sites. As with Google...it sometimes comes across a new website via websites already in its indexes.

Meta tags are good.

Robots.txt prevent google adsense and other things from working properly.

Just make sure you've submitted your site to as many search engines and that you've used Keyword Digger in BV to create your meta tags and you should be fine in a couple of weeks or so.

Good luck!

Originally Posted by rdarko

Another way that worked for me is to put as many links to popular sites on your website and do lots of link exchanges with popular sites. As with Google...it sometimes comes across a new website via websites already in its indexes.

Meta tags are good.

Robots.txt prevent google adsense and other things from working properly.

Just make sure you've submitted your site to as many search engines and that you've used Keyword Digger in BV to create your meta tags and you should be fine in a couple of weeks or so.

Good luck!

Thanks all!

Robot text is useful and has no bearing on googlebots. Googlebots search for content and are very slow. As for keywords use the keywords from your content page. Because your keyword must be in your content. If it is not you are penalized for it. Most important be honest with your website content and be sure it is relevant to your tiltle. I am no expert pet121

Just to add more for you all some look for urlist.tx, this a list of all your pages.

What you have to remeber it can take upto 2 months for your site to be shown in some serach engines, there is not 100% way of doing things other than using a service like VH offer

A friendly disagreement here, I agree a program like vh (voda hits a seo program)is likely a excellent product. I have had no opportunity to try it out so I can't really comment on it. But to get listed on major search engines without useing VH is just plain false. I am no expert, pet121

bottom line people is to do all of the above, there is no set rules, some search look for sitemaps, robot.txt, urlist.txt and xml file!

None of the above can hurt your listing!

I agree and i was wrong to say vh is a seo program because it is not. VH is a website submitter. pet121