Linked by Thom Holwerda on Fri 11th Dec 2009 18:35 UTC, submitted by google_ninja
Permalink for comment 399258
To read all comments associated with this story, please click here.
To read all comments associated with this story, please click here.
News
Linked by Thom Holwerda on 05/18/13 21:06 UTC
Linked by Thom Holwerda on 05/18/13 7:37 UTC
Linked by fran on 05/18/13 1:38 UTC
Linked by Thom Holwerda on 05/17/13 23:35 UTC, submitted by kragil
Linked by MOS6510 on 05/17/13 22:22 UTC
Linked by Thom Holwerda on 05/17/13 22:15 UTC, submitted by Tom
Linked by Thom Holwerda on 05/16/13 21:41 UTC
Linked by Thom Holwerda on 05/16/13 17:04 UTC
Linked by Thom Holwerda on 05/16/13 13:17 UTC
Linked by Thom Holwerda on 05/16/13 12:06 UTC
More News »
Sponsored Links



Member since:
2008-12-15
According to my weblogs for the multiple websites the company I work for hosts...
CUIL does not respect robots.txt.
In fact, from those same logs, it appears that CUIL uses robots.txt as a roadmap as to use to scrape a site.
Good luck using CUIL.