Announcement

Collapse
No announcement yet.

Proper location for robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Proper location for robots.txt

    I've read other posts on the subject, and I understand that unless I have a need to block the bots from crawling certain specific pages that a "blank" robots.txt file should be uploaded via the FTP manager in BV.

    Question 1) My question is that where should I upload it to on the server? To the public_html folder or should it be all the way in the top (root) folder. The one that is like this /.. ?

    Question 2) If I leave this robots.txt file blank, will the bots crawl only my site pages, or will they also crawl my server e-mail folders and my .cpanel, my .htpasswds, and .mozila folders?

    Thanks in advance for your help.

  • #2
    Re: Proper location for robots.txt

    Actually a blank robots text file will do NOTHING.

    You have to say which pages you dont want crawled.

    As a rule they will only travel your public html folder.

    You need one for each domain and it should be located in each domains folder

    if its the main domain then public_html/
    if its any other it will be public_html/domain name or subdomain name

    Bots do not have access to your cpanel as a rule as they cannot enter passwords. Thats why you have a public html folder.

    You will have to code in the pages you dont want them to access. Google for robots.txt and im sure you will come up with some tutorials on how to do this.

    Karen

    VodaHost

    Your Website People!
    1-302-283-3777 North America / International
    02036089024 / United Kingdom
    291916438 / Australia

    ------------------------

    Top 3 Best Sellers

    Web Hosting - Unlimited disk space & bandwidth.

    Reseller Hosting - Start your own web hosting business.

    Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


    Comment


    • #3
      Re: Proper location for robots.txt

      Beware also zipzap, when a robot cannot index your pages and contents,

      your pages cannot be archived for searching by the browsers.

      Dont expect to be No1 when you make a search for your website. Your website may not even be found!
      www.siapamoyanganda.com/
      Malaysian Family Tree Website From the
      State of Johor.

      HAPPY ARE THOSE WHO DREAM DREAMS AND ARE READY TO PAY THE PRICE TO MAKE THEM COME TRUE.

      Comment


      • #4
        Re: Proper location for robots.txt

        Thanks, Karen and Zuriatman,

        I really didn't have anything to put in there to shield from the bots, but in the Google Webmaster Tools, it was coming up that the bots were getting a 404 on my robots.txt file (it didn't exist anywhere) folder. I just put a blank robots.txt file in there just so I wouldn't see those 404s coming up on the Google webmaster tool page.

        A follow up question for Karen or anyone else - since she mentioned multiple domains. I want to set up another domain, but I don't understand how to structure my server file folders. Right now, everything is pretty much just dumped into my public_html folder, and that's OK for now, because I only have 1 domain. But what would my file folder structure look like if I wanted to run two separate domains on the server?

        Thanks in advance!

        Comment


        • #5
          Re: Proper location for robots.txt

          You shoould organize your public_html for ease of mantanence.

          Set up Folders and sub-folders so that you can store your files where it will be easy to delete and republish.

          Dont mix everything up like a thousand island salad.

          When you setup a New Domain [Add-on Domain] a new folder will be automatically created in your public_html with the add-on domain name.

          This folder represent your new website and you will have to setup new folders and sub-folders for this website.
          www.siapamoyanganda.com/
          Malaysian Family Tree Website From the
          State of Johor.

          HAPPY ARE THOSE WHO DREAM DREAMS AND ARE READY TO PAY THE PRICE TO MAKE THEM COME TRUE.

          Comment


          • #6
            Re: Proper location for robots.txt

            Zuriatman -
            So am I understanding you correctly that when I add a new domain, a new folder with that domain name will be automatically added to my public_html file? Next question, should my index page for the new domain also be created (by me) in that new folder? I was of the understanding that index pages have to be in the public_html folder to operate properly.

            Thanks in advance for your help,

            Matt

            Comment


            • #7
              Re: Proper location for robots.txt

              Matt..
              Only the index for your main domain goes in the public html folder.. after that they go in your addon domain folders....Which.. ARE in the public html/addon domain/

              Karen

              VodaHost

              Your Website People!
              1-302-283-3777 North America / International
              02036089024 / United Kingdom
              291916438 / Australia

              ------------------------

              Top 3 Best Sellers

              Web Hosting - Unlimited disk space & bandwidth.

              Reseller Hosting - Start your own web hosting business.

              Search Engine & Directory Submission - 300 directories + (Google,Yahoo,Bing)


              Comment

              Working...
              X