Bad Robot! · 3 May, 10:11 AM
I was trying to track down the IP address 213.206.76.79 because it’s been hammering my site, repeatedly requesting the same page over and over; for one post yesterday, it requested it 32 times in 33 minutes. Based on some research, that points back to herbert.groot.jebbink.nl and the WebImages application. I don’t post many images, so I have no idea why I’m being scraped so aggressively, but in any case there’s really no excuse for this kind of polling. WebImages/0.3 gets a mention Web Crawler Hall of Shame for not supporting Accept-Encoding: gzip; I’d call it out for simply pounding
on my site, even after explicitly disallowing it via robots.txt. Next step is an outright ban via .htaccess.
— Gordon Weakliem
Comment
Commenting is closed for this article.