Coding for No Search Engine Return (Full Version)

All Forums >> [Web Development] >> Search Engine Optimization and Web Business



Message

Larry M. -> Coding for No Search Engine Return (2/17/2006 15:48:34)

Hello Everyone,

I have 50 or so "private" pages on my real estate site server which are in place (without any meta keywords, except for the title tag) for my Buyer Clients only. Access is by and only known to the respective individual Buyer Client, e.g., www.carolinahomesrealty.com/jones.htm.

I want to positively prevent search engines from crawling or otherwise identifying these pages - my concern is Client privacy won't be protected if one or more pages are picked up by search engines.

With that said, does anyone know of a script or code I can add to the head section of each page that will stop any search engine bots (probably not the correct term)?

Thanks in advance [&:]



Mojo -> RE: Coding for No Search Engine Return (2/17/2006 16:10:39)

Spooky sells a login program - that's probably your best/easiest solution - unless you can code a simple one yourself. Which, it sounds like all you need is a simple login.

If all you need is to just protect the page I/we can write something simple for you in about 5 minutes.




womble -> RE: Coding for No Search Engine Return (2/17/2006 16:11:56)

<meta name="robots" content="noindex, nofollow" /> along with a robots.txt file should do it for law-abiding search engines.



Mojo -> RE: Coding for No Search Engine Return (2/17/2006 16:26:37)

quote:

I want to positively prevent search engines from crawling or otherwise identifying these pages



Larry M. -> RE: Coding for No Search Engine Return (2/17/2006 18:09:44)

quote:

<meta name="robots" content="noindex, nofollow" /> along with a robots.txt file should do it


Womble,

Sounds like what I'm after. However, what is a robots.txt file, how do I build it and where is it inserted when I get one?

Mojo, at the moment I'm going to try and keep it dirt simple but, if that doesn't work, I'll go with Spooky's Login.



Mojo -> RE: Coding for No Search Engine Return (2/17/2006 18:21:05)

Just so you know - the meta solution will NOT do what you asked.



BobbyDouglas -> RE: Coding for No Search Engine Return (2/17/2006 18:27:38)

quote:

ORIGINAL: Mojo

Just so you know - the meta solution will NOT do what you asked.

- I've noticed this too. Sometimes your website will get listed in SEs despite the fact you have a robots.txt file saying not to index anything at all. You have to either wait for the SE to come around and see the issue, or request your URL to be removed.



Mojo -> RE: Coding for No Search Engine Return (2/17/2006 18:33:22)

No to mention the rogue search bots that ignore robots.txt (there is no law saying they must obey them anyhow) and the scraper bots. Once a scraper has your content it will be uploaded to pages that the owner *ensures* will be indexed. So, your content will be openly available.

A 5 minute, simple login script (free) will eliminate that problem.



womble -> RE: Coding for No Search Engine Return (2/17/2006 19:25:32)

The meta solution's always worked okay for me, but then I may just have been lucky.

quote:

ORIGINAL: Larry M.

quote:

<meta name="robots" content="noindex, nofollow" /> along with a robots.txt file should do it


Womble,

Sounds like what I'm after. However, what is a robots.txt file, how do I build it and where is it inserted when I get one?


Basically a text file with instructions for SE bots telling them what they can and can't index. There's an article on it in the toutorials section on here if I remember rightly - goes in the root directrory.



Kitka -> RE: Coding for No Search Engine Return (2/17/2006 20:10:24)

I wouldn't go putting any hidden pages in a robots.txt file!

If the pages are not linked to any pages currently accessed by SE bots, placing the hidden file names in a robots.txt only makes them 100% findable and accessible. If you have ensured that there are no links anywhere for SE bots to follow, it'd be pretty amazingly lucky for them to find them accidently.

Anyone can view a robots.txt file, to look at the one for this site type in:

http://www.frontpagewebmaster.com/robots.txt

... and you'll be able to see exactly which files they don't want seen, or at least listed in the "good" SEs. Bad bots will either ignore a robots.txt or use it to instantly and easily find the files you don't want them to find.

I'd go with using the meta tag womble suggested and/or a simple login as mojo suggested.



Larry M. -> RE: Coding for No Search Engine Return (2/17/2006 21:05:45)

quote:

I'd go with using the meta tag womble suggested and/or a simple login as mojo suggested


Kitka,

A trifle confused here. MoJo says, to paraphrase, "the meta solution ... won't work". If there's a free login in script that everyone agrees will work, would you or MoJo (or anyone else) point me in the right direction, please?

Thanks!



Kitka -> RE: Coding for No Search Engine Return (2/17/2006 21:13:10)

Is the site hosted on Windows or Linux server?



coreybryant -> RE: Coding for No Search Engine Return (2/18/2006 9:13:08) 

<meta name="robots" content="noindex, nofollow" />
will not work usually for bad robots, but good robots will

Using a robots.txt - you can actually see the harm in doing that. Bad robots won't follow it and you have actually just told the entire world where your hidden files are

If you are on a UNIX server, htaccess would be one way. On IIS, there are a lot of scripts - and sometimes your control panel might even have a function. We have been working on a couple of free scripts as well - just takes time trying to please everyone :) - but you can also check out Spooky Login



Reflect -> RE: Coding for No Search Engine Return (2/18/2006 11:40:27)

I agree, if you REALLY want it private user a response login script as mentioned is the only solution in my mind.

Take care,

Brian



Taz -> RE: Coding for No Search Engine Return (2/18/2006 15:57:48)

Password Protect the directory at admin/server level?



Larry M. -> RE: Coding for No Search Engine Return (2/18/2006 18:51:56)

quote:

Is the site hosted on Windows or Linux server?


Kitka,

Right now the site is on an Apache Unix server. Because Microsoft is no longer supporting Unix with security patches, I'll have to move the site to a Windows 2003 server this September or possibly sooner if the CetLink server in Rock Hill, SC where my files are resident is hacked as others have been in the recent past [:@]

Because of the differing opinions here, it sounds like I may be Spooky's next customer. Was hoping get it done with a simple line of "no spidering" code. Ah, well.




Kitka -> RE: Coding for No Search Engine Return (2/18/2006 19:00:54)

quote:

it sounds like I may be Spooky's next customer.


Spooky Login only works on windoze servers, that is why I asked. It requires ASP.



Mojo -> RE: Coding for No Search Engine Return (2/18/2006 21:08:15)

quote:

Was hoping get it done with a simple line of "no spidering" code.


On windows (and I'm sure Unix), it can be done with a simple line - well, maybe 10 of them.



BobbyDouglas -> RE: Coding for No Search Engine Return (2/19/2006 1:00:06)

quote:

Right now the site is on an Apache Unix server.

- Just put some .htaccess password protection on it with the name of the protected area as "Please enter the number 1 for the username and password" and then have the user/pass be 1 to access this area. Usually this can be setup in your control panel.

quote:

Because Microsoft is no longer supporting Unix with security patches

- Care to explain a bit?



Page: [1]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
0.09375