robots.txt, DMCA, The Wayback Machine, and lawsuit

Posted By: g1smd (Staff) Posted On: 2007-Jul-27 01:34
The strange "hacking" case involving robots.txt, DMCA, The WayBack Machine and a specious lawsuit that could have had wide reaching implications had it succeeded... http://www.theregister.co.uk/2007/07/26/wayback_firm_suit/ Thankfully it failed.

Posted By: david68 () Posted On: 2007-Jul-27 14:21
At least the wayback machine obeys robots.txt (I checked their site and my site is blocked as I disallowed all bots not specified as being allowed). Several bots don't obey it and it's really annoying. I end up blocking them through htaccess regardless on their motive just because their aren't being polite.

Posted By: g1smd (Staff) Posted On: 2007-Jul-27 20:57
The point here is that for a short time, The WayBack Machine was not obeying robots.txt completely.

Posted By: Prowler (Staff) Posted On: 2007-Jul-28 09:18
>>The point here is that for a short time, The WayBack Machine was not obeying robots.txt completely. For some technical reasons. Good Post g1smd.

Posted By: Curt () Posted On: 2007-Aug-12 10:45
Wow! Guess all these bad bot spiders who don't play by the rules could find themselves in legal hot water some day for not obeying the robots.txt directives.