What page is the Robot looking for? (Full Version)

All Forums >> [Web Development] >> Microsoft FrontPage Help



Message

Peter -> What page is the Robot looking for? (5/15/2002 18:27:59)

My web site host gives me lots of useful info about the usage of my web site, including pages visited and pages not found.

Most of the 'pages not found' are because I have changed some of the page names, and the search engines have not yet caught up on what I have done - but I keep getting a message 'page not found' for the page:

www.edinphoto.org.uk/robot.txt.

It is true. No such page exists. All the pages on my site are htm, jpg or gif. But should I have also set up a robot.txt page to help the robots, and if so, what info should I include on it?

- Peter




caywind -> RE: What page is the Robot looking for? (5/15/2002 19:32:11)

robot.txt is a file that you can put in your web site that tells the search engines not to index the list of files that are in the file (robot.txt). what did I just say? anyway don't worry about it, it's not important unless you use it.

Fight the Spam! http://awebmasters.net/antispam.htm



Peter -> RE: What page is the Robot looking for? (5/15/2002 19:55:09)

Thanks, caywind.

I get the general idea. I'm happy for the robots to list any of the files they find on my site, so I'll not set up anything under robot.txt

(That's one thing less to think about! I find OutFront to be very helpful in providing quick answers to subjects like this where I have not been able to find the answer in my FrontPage guide books.)

Thank you.

- Peter




Reflect -> RE: What page is the Robot looking for? (5/16/2002 7:24:47)

If you just want to make the 404 go away create a robots.txt with notepad. Just open a blank notepad and do a save naming it robots.txt. Import it into the web on your hard drive. Then next time you publish it will go out into the root of your web.

Brian

Work hard, play fair, stay sane



Sandy Fairservice -> RE: What page is the Robot looking for? (5/28/2002 0:06:20)

A friend told me to put this bit of HTML in the header. I did, but I don't yet know if it worked. For what it's worth:
<meta name="robots" content="noindex">



Sandy Fairservice



caywind -> RE: What page is the Robot looking for? (5/28/2002 10:59:09)

look here...

http://www.robotstxt.org/wc/exclusion.html

the meta will keep some search engines from indexing the site. is that what you want? if not then just use the robots.txt procedure listed in the thread above...

Fight the Spam! http://awebmasters.net/antispam.htm



Peter -> RE: What page is the Robot looking for? (5/28/2002 17:43:40)

I've got what I need now, thanks to your help.

I'm happy for the robots to see my site. The only thing that was concerning me was the log of pages not found, which was giving higher numbers than I felt comfortable with - but I set up a blank page on notepad as you suggested.

The result: The robots are still looking at my site (as I want them to) and I am not now being told of lots of pages not found.

Thanks for your help.

- Peter




caywind -> RE: What page is the Robot looking for? (5/28/2002 21:26:46)

Always glad to be of service.

Fight the Spam! http://awebmasters.net/antispam.htm



Scotty -> RE: What page is the Robot looking for? (5/29/2002 1:04:21)

Wait a minute here! Did I miss something? Peter, I read your thread to say that the search engines are returning lots of errors, (due in part to your changing page names). Don't you want to redirect those searches to you home page or the page they were replaced with? That way you should get the benefit of searches for old and new. Several of the pros dealt with "redirects" on an earlier query of mine. When I find it, I'll list it here. Regards, Scotty

 



Reflect -> RE: What page is the Robot looking for? (5/29/2002 8:13:17)

Hi,

I would actualy make a sitemap. Then calll the host and have them set the sitemap as your default 404 page. That way if a spider/SE comes a knocking for a page that no longer exists they are fed some good and healthy spider food. If you go this root make sure to make the links straight text links (read no Javascript). Also give a good one sentence description using the main words/keywords from the target page.

Brian

Work hard, play fair, stay sane



Peter -> RE: What page is the Robot looking for? (5/29/2002 14:53:43)

Reply to Reflect:

Setting up a different 404 page. That sounds like a great idea, thanks. I'll speak to the web site host and find out how to do that.

I may create a page consisting of a brief note saying that I've changed the page names, followed by an extract from the top of my www.edinphoto.org.uk page.

- Peter




Texjd -> RE: What page is the Robot looking for? (5/29/2002 16:58:44)

Standard Operating Procedure: Always make a custom 404 page to return to site map or main menu. If people mistype a url or old or outdated links always will be handled. If you do a search in the root of your web site you can find the existing no info 404 and replace it.

Wasn't there going to be a FAQ somewhere on all the basics of creating a web site? I thought at one time somebody brought this up. This would be a good item to add since it's often overlooked.




Page: [1]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
0.0625