Is "robots" content="all" the best for me to use? (Full Version)

All Forums >> [Web Development] >> Search Engine Optimization and Web Business



Message

mr_tin -> Is "robots" content="all" the best for me to use? (2/23/2007 9:58:59)

Thanks everyone for all the help I have received on this website. I'm almost finished my first FP2003 website and I am now turning my attention to SEO.

If I want robots to see my site is the tag "robots" content="all" the best one to use?

Thanks - Martin



womble -> RE: Is "robots" content="all" the best for me to use? (2/23/2007 10:06:20)

<meta name="robots" content="index, follow" />



jurgen -> RE: Is "robots" content="all" the best for me to use? (2/23/2007 11:46:31)

You might want to use a robots.txt file as well.



mr_tin -> RE: Is "robots" content="all" the best for me to use? (2/23/2007 13:24:18)

What is a robots. txt file composed of?



womble -> RE: Is "robots" content="all" the best for me to use? (2/23/2007 14:21:43)

http://www.robotstxt.org/



mr_tin -> RE: Is "robots" content="all" the best for me to use? (2/24/2007 17:50:54)

Not sure I understand the robots.txt file thing. If I use Notepad to create a robots.txt file that has the following two lines in it saying:
User-Agent: *
Disallow:
What do I do with the txt file then? How do I get it on my index page? or do I put the two lines on the index page as meta tags or something?



jurgen -> RE: Is "robots" content="all" the best for me to use? (2/24/2007 20:49:17)

First, the robots.txt file should be in your root directory. It has nothing to do with you actual pages or html code.

In your initial post you ask for the meta tag "robots". Yes you want your site spidered, but maybe some of the files you don't want to show up in the search engine. For an example something like 
Disallow: /style/
Disallow: /images/

This would be the directories you instruct the robots NOT to index. You also can asign certain file as well.

With " User-Agent: * " you basically tell ALL spider to follow the rules of "disallow: whatever". You can disallow "user-agents" like googlebot, where this spider would not index your 'style' and 'image' directory.

You will have some control of certain crawler to do certain things. But on the contrary, "bad" and "nasty" crawlers just don't give a dung and do whatever they have been programed for. For the most part it will do the job.
If you put an empty "robots.txt" file in your root directory means you have noe restrictions to any robots, but at least you won't get the anouying "file not found" in your error log.... [8D]



mr_tin -> RE: Is "robots" content="all" the best for me to use? (3/4/2007 19:47:19)

I'm having problems with this concept...where is my root directory? I am already getting the error "File does not exist: /home/evanoff/public_html/robots.txt"

I just don't see the root directory in my published website, am I not seeing the forest because of the trees?

Here's my website - www.evanoff.ca

I would appreciate any help on this, I would like to get my site picked up by search engines.

Thanks - Martin



treetopsranch -> RE: Is "robots" content="all" the best for me to use? (3/4/2007 20:38:17)

Martin, are you using FP to publish or are you using a FTP program?

If you use FP just put the robots. txt file in the same directory as your index file on your home computer. When you use FP to publish it will put that file in the correct spot on the server.

If you use a FTP program just put it in the same directory as your index file on the server. (the main index, the one that opens your home page)

And don't forget to validate that robots.txt file here
http://tool.motoricerca.info/robots-checker.phtml



mr_tin -> RE: Is "robots" content="all" the best for me to use? (3/5/2007 18:40:18)

Thanks - that worked. I finally realized it just went in the root directory.

Martin <edit>Edited link</edit>



Page: [1]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
0.0625