Robot Attack !!!

 

Yes, in case you've noticed the site down... this would be the reason.

Robot Attack ! - well not the cool robot like the picture .. but a internet robot.  Some internet robots are good, and well behaved.  Search engines send out web-crawlers bots which slurp up data to be indexed.

They do it in reasonable bites (bytes).  

BUT THERE ARE EVIL ROBOTS - Ones which spawn a bajillion threads and try to download entire site in smallest amount of time possible.

On our little mrl site - it can quickly get overwhelmed with the horde of evil robots.

Here you can see a very very small snippet of the web server log

144.76.71.140 - - [04/Feb/2014:20:50:58 +0000] "GET /content/chatting-houston HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:51:11 +0000] "GET /content/some-pics-initial-prints HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:51:00 +0000] "GET /service/thingspeak HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:54:02 +0000] "GET /service/PointCloud HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:53:55 +0000] "GET /service/GUIService HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:54:23 +0000] "GET /service/Motor HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:54:13 +0000] "GET /blog/68 HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
144.76.71.140 - - [04/Feb/2014:20:51:31 +0000] "GET /content/easy-opencv-246-here-we-come HTTP/1.1" 503 2612 "-" "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)"
 
And here you can see the result....
 
[972354.298759] [ 8263]    48  8263   116099      319   0       0             0 httpd
[972354.298768] Out of memory: Kill process 30182 (httpd) score 25 or sacrifice child
[972354.298775] Killed process 30182 (httpd) total-vm:498932kB, anon-rss:3848kB, file-rss:80kB
 
That is the operating system saying.. HEY WEBSERVER YOU GOT WAY TOO MUCH MEMORY - YOU WILL BE KILLED !
 
Nooooooooo................. And then silence..............................  

 


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
GroG's picture

How to stop bad robots

How to stop bad robots :

GroG's picture

KMC

KMC suggested

GroG's picture

Excellent EC2 Micro HTTP

Excellent EC2 Micro HTTP Adjustments

http://imperialwicket.com/tuning-apache-for-a-low-memory-server-like-aws...