Don't forget the robot txt file!

[limebrook1]

Just a little reminder - there I was, new site up, Google had visited but hadn't looked at much. Why?

Yet again I had forgotten the robot txt file! The search engine bots really love these, so don't do what I did and forget it!

[Bethers]

Re: Don't forget the robot txt file!

Personally,
I don't bother with it - Google has been on my site about every other day. Content is really king - Just like the revisit tag- it means nothing if there is no new content - they will NOT come back in 5 days just cuz you say so.

Now, that doesn't mean the tag will hurt - but don't give most of these tags too much credit. Now, if you have anything you don't want the se's to spider - then this tag is a must. This is were that specific information needs to go.

[limebrook1]

Re: Don't forget the robot txt file!

I hear what you say, Bethers, but I have found my sites have suddenly come to the attention of the search engines immediately I add this file. They seem to take it as a 'come and visit me' signal, and I am not just talking about Google.

Before I put up the file they seem to stay away! I know you have more experience than me, so perhaps it is just a coincidence!

[Bethers]

Re: Don't forget the robot txt file!

They can't know you added the file before they visit you - they don't have a sixth sense. So, adding the file, then they visited, is just coincidence.

They also have a "preset" time they usually come back and visit - they determine this based on their past visits. If when they come, they see changes, new content, etc - they put you down to visit more often. If it looks identical for a couple visits - they'll lengthen the time between visits - and no matter what you put for when you want them to revisit - they'll just ignore it.

So, keeping your sites updated regularly is vital. Someplace like this forum will be spidered daily at least because of all the new content. A site that never changes, might only get visited once a month. That is not always bad - however when you make changes it will take the spiders longer before they do see them.

[Andy128]

Re: Don't forget the robot txt file!

Can anyone post the examples of good meta spider tags?

Cheers,

Andy

[Bethers]

Re: Don't forget the robot txt file!

Not using them are good ones - as they aren't necessary.

[Robert]

Re: Don't forget the robot txt file!

Can anyone post the examples of good meta spider tags?

Cheers,

Andy

I've been waiting to find some interesting articles on the algorithms of search engine spiders to help give me a better understanding of what process they go through when they reach a site and what the "hotspots" are.

I lack alot of content on my site at the moment, but I've been lining it up and taking a gamble on my keyword strategy that I've been working on for a few month's now... Relevancy. I searched Google using: SE+spiders+algorithm

The top of the results was an article. A new algorithm is being developed by Google and the 26yr old Australian who sold it them, called "Orion".

http://www.websitesource.com/news/orion-algorithm.shtml

You know, all they have to really say is that they want a spider to be on the same wavelength as us when we make a search query.

One thing that interested me, though, was when they metioned how this new algorithm will be able to offer alternative search querys from the users' based on relevancy. (Again, this is similar to the 'Did you mean...' but more .....well, read it for yourselves. OH, and read it again, between the lines. Sounds to me like it's a clever way of taking control of what pages they want users to see by creating a new algorithm that will most likely scramble the ranks, increasing demand for paid rank placement, professional SEO services, and more. Then again, this is only my perception, but think about how this will affect small to medium websites, companies, etc. that rely on online traffic. If you plan is only to steadily increase traffic, then no problem. But if you plan on actively competing for the ranks, then it sounds to me that some major $$$$'s will be needed and Google will be holding the plate.

[Bethers]

Re: Don't forget the robot txt file!

[quote]You know, all they have to really say is that they want a spider to be on the same wavelength as us when we make a search query.[quote]
And that's it in a nutshell.

They do not want people to "fool" them into getting into the results, tho - so they are attempting to weed out all the ways people can do that. Which is one reason why meta tags like keywords and revisit are not worth squat anymore. And why link trading has lost lots of value - but one-way inbound links are still important.

Make the most professional site - with no typos, spelling and grammar checked - easy to navigate - using the terms people are using to search for you - write articles as you can - do press releases as you legitimately can - find your niche - blog or find forums with your topic interest - do it all without trying to fool anyone.

There are SEO methods other than mine that will work - and there are many out there touting methods that are worthless and just going to take money - but, whoever you listen to - don't try to mix 2 or 3 ways - as you'll probably do more harm than good.

The SE's as stated above - want nothing more than to deliver the RIGHT response to the terms entered into them.

[Robert]

Re: Don't forget the robot txt file!

Absolutely Bethers, and I was hoping you would post after me with that exact response...I was being rushed in the middle and had to cut short.

Thank you kindly.

:)

[Vasili]

Re: Don't forget the robot txt file!

Focus on the "Perfect Page" concept of precision website construct, and you will achieve the desired results......

Google changes it's "al's" periodically, and yes, it does have a lot to do with their merging activity (ads, programs, etc.) and relevancy.
Based on overall relevancy (content IS king), even adding stupid little bot tags count to the "Perfect Page" theory......and that is why all the worthy
SEO experts are zeroing in on 'perfecting' hard coding even in templated sites using php (carts especially, with their inherent haphazardly written code), Fusion (are there still some out there?? LOL), and especially Dream. Bone up on what is going on in the .net world, and how database config and morphing is playing a part if you really want to get a glimpse of web/internet tweaking....SEO is becoming a type of gauze to break through to get to G2 of online functionality!

Pretty much everything else can be accomplished ourselves, if we "get it" and pay attention to all the minutia of proper construct.

Unfortunately, I believe most will never "get it" and struggle desperately with the "relevancy" concept completely, especially with regard to Content.

[Robert]

Re: Don't forget the robot txt file!

Vasili, the way you put my thoughts into words is phenominal. I also wanted to make note of how "RELEVANCY" is definitely and QUICKLY becoming the term of the hour.

A previous post I made with a referrence to an article about Google and their newly acquired algorithm, "Orion", appears to be focused around relevancy issues concerning pages, links, and SEspiders. I also said that alot of sites will dramatically be affected by this. Check this out:

------
*copied from a ContourCandy Affiliate Newsletter I recieved:
Newsletter for 12/01/06

SEO TIP
Google Dance, check your rankings. Because Google's results change daily, it's difficult to say we saw a Google Dance late November, but we did see an update. Search Engine Result Positions("SERPs") have changed. Be sure to analyze your keywords for movement. It seems like sites with an excessive amount of unnecessary links, cloaked text and duplicate content have been hit.

http://www.contourcandy.com

------
I wonder if parts of "Orion" are already implemented into their algorithm?
hmmm. food for thought.

Even so, this is why I have been carefully planning and designing my future content around relevant content first, then optimizing the keyword structure. I figure, it is easier to change your keywords on the fly rather than to have to change your content.

Oh, and Vasili, thanks man. I can't begin to tell you how your posts clarify moments in my thought process. Your communicate with words in the same manner as my mind produces thoughts. Keep up the great work!

[Bethers]

Re: Don't forget the robot txt file!

Relevancy is NOT new and has ALWAYS been the basis of seach engines listing their findings on searches. What we are constantly seeing are the se's refining their algorhythms - mainly because people keep trying to "fool" them. They are not stupid.

So, if people would simple remember to do things properly to begin with - the se's wouldn't have to keep having to refine to catch the "cheats". Although for those of us who don't fall for tactics that aren't dealing in relevancy and clean codes, etc - we actually are helped when the idjits get caught.

And I say idjits with some compassion for some of them. Keywords were one of the first things to lose value - because people didn't understand that you can't select your keywords by putting them in the meta and then thinking you don't have to use them in any manner - they have to be part of the site - whether on page or off.

But, my point here, is that relevancy is not a new phenomena - it's ALWAYS been important. And content is king.

[Vasili]

Re: Don't forget the robot txt file!

As always, the "last word" is more like an echo, with something quite "core" being glossed over ..... rather than repeat and reinforce the importance of relevancy (as I had tried to illustrate using no less than 3 contrasts), it would be more beneficial to zero in on "Perfect Page" methodology and how far above the radar it is being purposely held!

It is a concept that can be brought right down to the solid ground in VodaLand (as we have the best tools available), and can dubiously be put to good use before the masses pick up on it......(it has been discussed for more than 6 months, but only recently alluded to in a private Google inter-company memo).

So much for cognizant expertise, huh?
THINK. APPLY. RE-THINK, and REFINE.
Take the time to read, to understand what you read, and to define your understandigs in your undertakings....to the best of your ability! (and DO plan to update content regularly....the basis for a dynamic utility being a part of your site :: boards, "news", photos, articles which can add to intra-site navigation and published relevancy, etc. -- but keep in mind to ADD and not remove content)

(Orion, Yahoo, and 3 other concerns each have been battling Google for some time about the value/accuracy of their searches if the stes they present results from are less than "perfect" and therefore "unvalidated".....all about construct, in the purest logical form).