RE: Google Page Rank and Search Results Position? (Full Version)

All Forums >> [Web Development] >> Search Engine Optimization and Web Business



Message

Mike_R -> RE: Google Page Rank and Search Results Position? (9/16/2003 14:46:30)

I see. I like the idea--it make's sense. As Brian suggests, I may be too late. I wish I had known in the beginning. Oh well, I guess I will have to design another site someday.

Incidentally, do you know why only the first page in a section/folder gets ranked by Google? I was curious about that. And, does it harm or affect your ranking on other pages when pages like your privacy policy get indexed?

Mike



Reflect -> RE: Google Page Rank and Search Results Position? (9/16/2003 15:21:59)

quote:

As Brian suggests, I may be too late.


Mike I was not referring to you on this one, sorry if it sounded like a slam. It was not intended to be so.

quote:

Incidentally, do you know why only the first page in a section/folder gets ranked by Google?


This was only noticed, at least by me, for directories that I excluded via robots.txt file or META no index. normally all files get indexed and ranked.

quote:

And, does it harm or affect your ranking on other pages when pages like your privacy policy get indexed?


None what so ever. I just figure why pass PR or why tax the bot/spider?


Sorry for the confusion that I made Mike [;)].

Brian



Mike_R -> RE: Google Page Rank and Search Results Position? (9/16/2003 19:03:42)

Brian, I didn't take it as a slam at all. I know that in the learning process of developing my first site, I made some bad calls in defining priorities. I smile at it now and actually feel proud that I have learned as much as I have. Nothing like a good screw-up to teach a good, and permanent lesson. The next time I build a site, I am going to be able to do it in 1/3 the time I built this one. And I owe most of that to the expert advice received from you and others in the Outfront Forums. You guys have really saved my sanity a few times.

Mike



coreybryant -> RE: Google Page Rank and Search Results Position? (9/17/2003 11:11:57)

Andrew, when you say:
quote:

My motivation is to prevent the bot from becoming lost or timing out before it can get some real data. I also block the FP directories and cgi-bin:

Do the bots actually find some of those web pages if they are not actually linked? I could possibly see the cgi-bin & _private but what about _vti_cnf - since I do not think those folders are actually referenced anywhere in the FP coding?

I added a lot to my robots page (to help keep the spammers & e-mail harvesters out) so I am interested in knowing more about this actually as well. Thank you!



awasson -> RE: Google Page Rank and Search Results Position? (9/19/2003 11:14:47)

Hi Corey,

Sorry about my lack of response. This silly work thing gets in the way sometimes.

The calendar is linked through my navigation section, under application development services, and the bots would get in there and more than likely time out while indexing thoudands of pages of dynamic content, or expire trying to.

This also meant that in the case of google I would get visited by the google bot about once a week if I was lucky.

Since I blocked the calendar, I get visits almost every day without fail and my results are much better in the ranks.

I also blocked the _vti_cnf and other private folders just to be safe. It just takes a minute. If you're on Unix you can make adjustments that prevent certain user agents access from chosen directories by adjusting your .htaccess file.

I think it would be possible to do the same thing with ASP on a windows server to run a small script at the top of every page that loops through a list and if it finds a known spam agent it dumps them into an empty directory and also redirects good bots to the places you want them.

Cheers,
Andrew



pageoneresults -> RE: Google Page Rank and Search Results Position? (9/22/2003 10:11:40)

Any folders in your web that begin with an _ (underscore) are automatically off limits to any spiders and/or users. Typically if you browse to those files with the underscore in them, you will get a username/password prompt so there really is no need to clutter your robots.txt file with them. I always put the _private in there just to be on the safe side, the others are of no importance.

If you are that concerned about filtering or directing PR within your site, you need to give careful consideration to your linking structure. You can easily block certain pages from being indexed by using the robots.txt file or a META Robots tag.



Mike_R -> RE: Google Page Rank and Search Results Position? (9/23/2003 23:54:03)

quote:

If you are that concerned about filtering or directing PR within your site, you need to give careful consideration to your linking structure. You can easily block certain pages from being indexed by using the robots.txt file or a META Robots tag.


Pageone, you got me thinking with that comment. If I list my "Includes" folder in my robots.txt file, will that keep the spiders from indexing the individual pages (header, footer, menu, sidebar, e.g.), or will it keep the spider from indexing all pages that use those pages--i.e. all of my content pages?

Mike R



Reflect -> RE: Google Page Rank and Search Results Position? (9/24/2003 7:45:47)

quote:

If I list my "Includes" folder in my robots.txt file, will that keep the spiders from indexing the individual pages (header, footer, menu, sidebar, e.g.), or will it keep the spider from indexing all pages that use those pages--i.e. all of my content pages?


I believe that it would still read the content as a spider/bot sees the include inline, not as a seperate page (the include has been rendered like you viewing the page with a browser). Now if it is just trying to crawl the directory then I would think it would look at the robots.txt file and bypass.

I say this as I have included my "includes" directory in my robots.txt on my first site. The bots however showed pages in its SERPs that could have only been reached via my "include" pages navigation.

Brian



flintops -> RE: Google Page Rank and Search Results Position? (2/16/2004 14:18:50)

Hi guys,

Also remember in in text links too. As the Google technology gets more sophisticated, then links pages will carry less value.

Kind regards,

Paul



Page: <<   < prev  1 [2]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
0.0625