|
| |
|
|
TODDMAN
Posts: 64
Joined: 6/13/2001
From: Derby, KS
Status: offline
 |
Google Bot - 2/22/2007 23:26:27
I received this email from my hosting provider dreamhost.com.
“We have noticed that your site (gallery.toddman.org) is getting hit hard
by the googlebot, google uses this to search the content of the site to
list it on google.com, this is maxing out the connections on the server
and causing load to go up. In an effort to bring the load down we had to
block the googlebot IP address to stop it from accessing your site, if
you don't care about google indexing your site there is no further action
that needs to be taken, if you do want google to index your site remove
the IP address from the .htaccess file that we placed in your domain
directory. and go through Google help site here
http://www.google.com/support/webmasters/bin/topic.py?topic=8843 to fine
out how to you lower the amount of crawling that the google bot does,
this will help keep the load on the server down.”
I followed the link to:
“What can I do if Google is creating too high a load on my server?
If Google is causing excessive strain on your servers and you'd like us to slow the rate at which Googlebot crawls your site, please let us know. In your message, please include a text snippet from your most recent weblog that lists Googlebot. Also, please confirm your request by creating a forgoogl.html page on your site and sending us the URL. We will then pass your request on to our engineers. “
Only problem I can’t get into my logs, waiting on my host to give me permission.
Below is a copy of the file inserted by my host.
“.htaccess
order allow,deny
deny from 66.249.66.143
allow from all”
I moved my gallery from Infinology.net to dreamhost.com, because Infinology has a 20 Meg limit on MySQL, $0.15 per meg over the limit, in which I was exceeding weekly.
Has anybody else had this problem with their hosting provider? Banning a bot from their site?
OBTW, I’m still building the site!!!
Thanks, Toddman
www.toddman.org
|
|
|
|
Thomas Brunt
Posts: 6109
Joined: 6/6/1998
From: St. Matthews SC USA
Status: offline
|
RE: Google Bot - 3/5/2007 8:58:13
Not sure if I'm talking about the same thing.
I have seen a similar problem with a couple of clients, but it was not the Google spidering that was the problem. It was the Google searches that returned their images. The image downloads were eating up all the allocated bandwidth, and they were not providing the site owners with anything useful.
The robots.txt file was helpful, but the first thing we did was to change the file names of all the images and all of the pages displaying the images. This seemed to do the trick.
t
| |
|
|
|
BobbyDouglas
Posts: 5456
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Google Bot - 3/5/2007 14:53:44
Your website loads incredibly slow. Places like dreamhost, 1&1, ipowerweb, all have fancy looking packages for hosting, at great prices. The problem these hosts run into, is that there are just too many clients on a single server. Everything runs slower, and they have many more clients to deal with. What happens when a severe issue rises? They have 5,000 people down, many calling in and wasting the host's time. Imagine if there were only 2,500 people on the server? Although their prices are low, they still have to make up for it by loading up the servers with lots of clients.
Your host should have told you what was causing the loads, since they didn't, you are now sitting with your website while googlebot is being completely banned from the entire site. Luckily, they only banned a single IP address. Any idea how much bandwidth is actually being used from googlebot? Do you have any type of stats software on your site?
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
|
|
onekgguy
Posts: 5
Joined: 2/28/2006
Status: offline
|
RE: Google Bot - 3/5/2007 16:06:32
Yes, your site loads very slow...I gave up on it. Web space is quite cheap and good companies are out there who can handle your traffic without limiting access from the googlebots. I've never heard of that happening...consider finding a host better able to meet your demands. I would think you would want the bots indexing your site so people can find your content through a Google search.
Kevin g
| |
|
|
|
TODDMAN
Posts: 64
Joined: 6/13/2001
From: Derby, KS
Status: offline
|
RE: Google Bot - 3/5/2007 16:14:08
quote:
Your host should have told you what was causing the loads,
They did, my gallery.
quote:
now sitting with your website while googlebot is being completely banned from the entire site.
So much for being SEO friendly!
quote:
Luckily, they only banned a single IP address.
How do you know this? I removed the .htaccess file & placed a robots.txt file in its place, but that didn't help a different google bot came by & ignored the robots.txt file. Dreamhost had another cow & without telling me added another .htaccess file banning goggle. In the last 7 days Dreamhost had a magor melt down & that didn't help much. Currently looking to move again to puppy power hosting. I know the owners!
quote:
Any idea how much bandwidth is actually being used from googlebot? Do you have any type of stats software on your site?
Next to none, with everything I have going 4 domains & 4 subs, my bandwidth was .02GB out of 2TB limit. The stats software is analog, I can post it if you like.
| |
|
|
|
Kitka
Posts: 2515
Joined: 1/31/2002
From: Australia
Status: offline
|
RE: Google Bot - 3/5/2007 16:21:50
quote:
but that didn't help a different google bot came by & ignored the robots.txt file.
It is extremely rare (i.e. almost unknown) for a real Googlebot to ignore robots.txt. So it was either a fake Googlebot (many dodgy bots spoof the Googlebot UA) or it might be because your current robots.txt gives all bots carte blanche to take anything and everything:
User-agent: *
Disallow:
If you want to ban all bots you need:
User-agent: *
Disallow: /
_____________________________
Kitka
**It is impossible to make anything foolproof because fools are so ingenious.**
| |
|
|
|
BobbyDouglas
Posts: 5456
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Google Bot - 3/5/2007 17:19:59
quote:
They did, my gallery.
- There are many parts to the gallery. Most likely it was the images, but you need to know for sure, and your stats software should tell you.
quote:
How do you know this?
- It shows it here:
quote:
Below is a copy of the file inserted by my host.
“.htaccess
order allow,deny
deny from 66.249.66.143
allow from all”
The deny from 66.249.66.143 line is the Googlebot IP address.
quote:
Currently looking to move again to puppy power hosting. I know the owners!
- You've moved quite a bit already, make sure they provide a good service before you switch again. As long as you are NOT hosting a personal website, then you want to avoid these cheapo hosts. Expect to pay around $10/month for a low amount of space, from a place that offers good support.
quote:
Next to none, with everything I have going 4 domains & 4 subs, my bandwidth was .02GB out of 2TB limit.
- So Googlebot is "maxing out the connections on the server and causing load to go up" and you have only used .02gb of bandwidth? Are you sure that wasn't your bandwidth for March? Sounds very very low to cause issues with even a cheap host's server.
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
|
|
BobbyDouglas
Posts: 5456
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Google Bot - 3/5/2007 18:11:33
quote:
The bandwidth is for Feb.
- You should ask them how one IP address was able to make such an impact on the server, yet use less than .02gb of bandwidth. It would be interesting to see their response.
Websites that have issues with the Googlebot, are ones that are pushing out bandwidth in the amounts of multiple GBs.
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
|
|
TODDMAN
Posts: 64
Joined: 6/13/2001
From: Derby, KS
Status: offline
|
RE: Google Bot - 3/6/2007 12:07:11
Here are my stats for Feb.
Web Server Statistics for toddman.org
Analyzed requests from Thu, Feb 01 2007 at 3:36 PM to Wed, Feb 28 2007 at 8:06 PM (27.19 days).
Successful requests: 1,704 (491)
Average successful requests per day: 60 (70)
Successful requests for pages: 1,462 (364)
Average successful requests for pages per day: 52 (51)
Failed requests: 44 (5)
Redirected requests: 4 (0)
Distinct files requested: 108 (8)
Distinct hosts served: 44 (4)
Data transferred: 3.17 megabytes (1.72 megabytes)
Average data transferred per day: 116.06 kilobytes (251.33 kilobytes)
Web Server Statistics for gallery.toddman.org
Successful requests: 38,698 (9,932)
Average successful requests per day: 1,391 (1,418)
Successful requests for pages: 30,424 (8,047)
Average successful requests for pages per day: 1,094 (1,149)
Failed requests: 459 (5)
Redirected requests: 8 (1)
Distinct files requested: 15,199 (324)
Distinct hosts served: 50 (7)
Data transferred: 106.24 megabytes (53.69 megabytes)
Average data transferred per day: 3.82 megabytes (7.67 megabytes)
Web Server Statistics for sportz.toddman.org
Successful requests: 3,495 (358)
Average successful requests per day: 128 (51)
Successful requests for pages: 922 (96)
Average successful requests for pages per day: 33 (13)
Failed requests: 4 (0)
Redirected requests: 3 (0)
Distinct files requested: 446 (25)
Distinct hosts served: 48 (3)
Data transferred: 9.00 megabytes (1.13 megabytes)
Average data transferred per day: 339.01 kilobytes (165.57 kilobytes)
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
