Comments on: Bad MSN

By: Ploum

Ploum — Thu, 01 Jan 1970 00:00:00 +0000

It seems that MSNbot access all IP that appear in a MSN chat.
You are not the first I hear about in this case.

By: Jeremy Nickurak

Jeremy Nickurak — Thu, 01 Jan 1970 00:00:00 +0000

Outlook/Explorer/Messenger watching URL’s discussed/browsed as a source for msn indexing targets?

By: acid

acid — Thu, 01 Jan 1970 00:00:00 +0000

Maybe someone typed this URL in his search/URL toolbar and its f*cking browser did try to search it with MSN search engine ?

Since the search engine did not find it, it tried (maybe later) to check it...

By: Adam Rosi-Kessel

Adam Rosi-Kessel — Thu, 01 Jan 1970 00:00:00 +0000

Interesting suggestions. It should be possible to test the MSN chat hypothesis–create a “trap” for the MSNBot with a URL that is only used in MSN chat. Has anyone done that? I would think there might be a minor privacy uproar if MSN is monitoring chats for ideas of what to spider. hotmail would be even worse.

By: Thomas

Thomas — Thu, 01 Jan 1970 00:00:00 +0000

Your second point isn't possible if you know every other entry in the log.
Are you 100% sure that nobody that knows the URL ever linked to that page? Probably using a bookmarking service or something similar.
The automatic msnsearch explanation seems reasonable too.

By: Foo

Foo — Thu, 01 Jan 1970 00:00:00 +0000

Why would it be so unlikely that Microsoft snoops on hotmail mails? AFAIK they send MSNBot to urls that appear in msn messenger messages.

By: Pharao

Pharao — Thu, 01 Jan 1970 00:00:00 +0000

you can keep the bot away from your page with some rewrite rules in the apache config.
There was a link in http://www.linux-web.de/artikel/5031/hilight,msnbot/kein-MSNBot-mehr-.html
but 1) it is a german forum and 2) the link to the config seems to be offline… maybe “michael” would upload it again if you ask him…

By: Martijn Vermaat

Martijn Vermaat — Thu, 01 Jan 1970 00:00:00 +0000

Visiting links appearing in MSN Messenger messages would be very bad practice in my opinion.

As far as honouring the robots.txt directives concerns, my experience with MSNbot is that it is actually doing this right, so I'm suprised you say it ignores the robots.txt on your server.

By: Adam Rosi-Kessel

Adam Rosi-Kessel — Thu, 01 Jan 1970 00:00:00 +0000

Actually, on closer inspection, I discovered that my htaccess file was blocking access even to robots.txt to robots, so the MSNBot likely got an “access denied” error in trying to request the robots.txt file. I suppose the behavior is not quite as bad, then. I’ve fixed it now.

By: Tuukka Hastrup

Tuukka Hastrup — Thu, 01 Jan 1970 00:00:00 +0000

Perhaps you already thought about this, but if you have outgoing links from that secret page, the visitors’ browsers have leaked the URI to the target sites as the http referrer address.