PDA

View Full Version : help with robots



pet121
08-31-2005, 07:07 PM
For quite sometime I have been getting this error message

[Wed Aug 31 10:43:58 2005] [error] [client 213.180.216.253] File does not exist: /home/onlineg/public_html/robots.txt

I do have robot text in my source but robots, crawls whatever you call it, are having a hard time finding my page. Does it have something to do with me uploading my files to public_html? please help I am begging. pet121

Maddog
09-01-2005, 09:00 AM
Search engines will look in your root domain for a special file named "robots.txt" (http://www.mydomain.com/robots.txt). The file tells the robot (spider) which files it may spider (download). This system is called, The Robots Exclusion Standard.

Maddog
09-01-2005, 09:08 AM
check out

http://www.webtoolcentral.com

pet121
09-02-2005, 12:22 AM
I am getting there slowly. This is turnning out to be no hobby thats for sure. My site well its not even a quarter finished but put it on the net anyway. I keep on getting sidetracked on learnning how to promote your site. I appreciate your response and will look into it. Some robots grab some don't. I am doing very well on google and yahoo. 35000 hits last month not to shabby for a quarter of a site. Thanks again pet121

djdiamonds
09-02-2005, 09:10 PM
what is robot text and how do you put it on your website? i want to optimise my searches on search engines for my site www.davidjoffediamonds.com (http://www.davidjoffediamonds.com) and i dont know why everyone keeps mentioning this robot text!! PLEASE HELP

Maddog
09-02-2005, 09:23 PM
what is robot text and how do you put it on your website? i want to optimise my searches on search engines for my site www.davidjoffediamonds.com (http://www.davidjoffediamonds.com) and i dont know why everyone keeps mentioning this robot text!! PLEASE HELP

david

go here

http://www.webtoolcentral.com/webmaster/tools/

this site will help you make your "robot.txt" and explain all about it, also there some other great tools for helping to get your site listed/ranked

any problem just ask and I will try and help

Steven.H
09-02-2005, 09:29 PM
david

go here

http://www.webtoolcentral.com/webmaster/tools/

this site will help you make your "robot.txt" and explain all about it, also there some other great tools for helping to get your site listed/ranked

any problem just ask and I will try and help

Thanks for the website.

djdiamonds
09-02-2005, 09:56 PM
thanks very much for you help - i didnt quite understand but i think i did it right :~

Maddog
09-02-2005, 10:22 PM
thanks very much for you help - i didnt quite understand but i think i did it right :~

Not much to do other than copy to note pad save as robot.txt and the upload to you web site using FTP,

Also have a look at your meta tags, get this forum on changes to make.

The other part to help search bots is to have a sitemap, go to

http://www.blocklayer.com/googlesitemap/Default.aspx

will help you make what you need

djdiamonds
09-02-2005, 10:51 PM
thanks so much - you make it all seem so easy!! im almost certain i created a sitemap and i managed to create meta tags!! (hooray)

paulb
09-03-2005, 08:45 PM
Hi Maddog,

Just used the free tool for robots file, this is what came back, it doesn't seem a lot, I expected a lot of code,

Paul

Here is what your robots.txt file should look like;

# Robots.txt file created by http://www.webtoolcentral.com
# For domain: http://www.superfastincome.com

# All robots will spider the domain
User-agent: *
Disallow:

Maddog
09-03-2005, 11:21 PM
Know what you mean Paul not alot to it, looks ok to me.

pet121
09-04-2005, 01:59 AM
wouldn't the below robot text disallow robot to enter?



# Robots.txt file created by http://www.webtoolcentral.com
# For domain: http://www.superfastincome.com

# All robots will spider the domain
User-agent: *
Disallow

I don't want to get it wrong, big penalties. My ranking are going well without it. I am pr3, but futher optimization is needed. thanks pet121

Maddog
09-04-2005, 10:40 AM
the * is not a wild card to disallow

User-agent: *

here you would put the bots disallow, my understanding is blank means allow all

pet121
09-04-2005, 06:54 PM
Okay I understand but I am still getting these error messages. Out of 72 robots that visited my site only 17% actually found it. Its just weird. Thanks anyway pet121

Maddog
09-04-2005, 06:55 PM
who do you know 72 visits and only 17% found it?

Maddog
09-04-2005, 07:05 PM
Okay I understand but I am still getting these error messages. Out of 72 robots that visited my site only 17% actually found it. Its just weird. Thanks anyway pet121

just been looking at your site pet,and code, you need to name each page and page the meta tag in each page, remeber can take upto 60 day for search companys to update there sites

pet121
09-05-2005, 12:21 AM
I will double check the source again and make sure each page is named. thanks pet121

steve_w
09-08-2005, 10:17 PM
Maddog,

I have visited webtoolscentral and tried their metatag analyser thingy; and, horror of horrors, it tells me 'error, your site cannot be found.' I am completely perplexed by the whole search engine, meta-tag, description thing... I can access my site by typing the name followed by .com, but on google, aol and msn (all I have available), if I simply type the name of my site, it is also not found. In desperation I have paid 3.49 to eBiz1000 to submit my site to some search engines for a month - was this a sensible thing to do? I found a link to them on webtoolscentral and it didn't seem like a lot to lose but now I'm beginning to wonder. Earlier in the day I subscribed to a free service which offers subscription to 40 engines for nothing and almost immediately I had a couple of spiders on my site, whereas there has been no sign of a spider in the three hours since I used the paid service. Crikey I'm confused. Why should people have to pay to get their site listed on search engines? It seems implicitly wrong to me. (On the Vodahost tutorial there is a heading related to 'submit to search engines,' but that cannot be found in the control panel as shown in the video tutorial - I wonder why... Anyway, anything related to search engines / getting listed would be greatly appreciated, as I'm becoming fed up of constant spamming to attract visitors to my site. Extremely jealous of the person whose site is only one quarter finished and attracting thousands of hits - luck?

pet121
09-09-2005, 06:04 AM
Its funny you asking for help? Currently I am listed on all the major search engines. On yahoo page3, msn page1, alta vista page1 plus I rank high all over the world. How did I do it no idea? I noticed the your sites are in flash I heard could be a problem for robots, look into it. I will probably get into trouble for this but you need a good seo program, voda hits is one of them, haven't had a chance to try it out yet. But in the mean time down load the free trial version of web ceo but be sure you know what you are doing. Web ceo will automatically submit your site once. But on a ongoing basis you can see were you rank on the internet. If fact there are alot of freetrial seo programs out there. Market your site its not easy but keep plugging away at it. For instance web ceo reported that i had 298 links on yahoo. Another way is creating back links. Some forums are crawled by robots and if you leave your website address on a thread it will pick it up and post on say google. There is so much more information. The bottom line it takes some hard work. pet121



p.s do not pay anything to get good rankings or listed!

Maddog
09-09-2005, 07:35 AM
Steve W whats your web address (url to your site)

As pet121 said you do have to keep pluging at it, are you new to BV? if you are VH will submit free your site, just open a support ticket, you do have to rememebr it can take wait for search companys a while to get around to your site and index it. as for paying for it, well some people are good some are not.

pet121
09-09-2005, 02:55 PM
Bv as far as I know only submits to a limited amount of sites. As for paying to have your site listed and get good links and ranking forget it. The search engines who require cash to submit are not really worth it. Eventually the paid search engines who charge will include your site anyway. This will be do to the preasure of being every where else on the net. I have proof of this happenning. For example, ask jeeves search engine is a paid for inclusion, but I am listed. The funny thing is I just can't figure out how I did it? I am still learnning from other sources however. In less than three months I am listed on yahoo page 3 and have 198 links on yahoo, how, I don't know. thanks pet121.


P.S maddog my prevous post on this thread was meant for Steve. My wife insisted last night we have a few drinks.