Apache hung

[chase]

Looks like apache hung yesterday. The max num of httpd processes were running and non of them were responding. Looking at a stack track it seems that all of them got stuck in the apc module (php cache). I've changed some of the settings for apc, hopefully it won't happen again.

I've also setup a pingdom.com account so I should be notified within a couple of minutes of wiki.osdev.org going down.

Code: Select all

# pstack 16655
#0  0xf57fe416 in __kernel_vsyscall ()
#1  0xb741c0d9 in __lll_lock_wait () from /lib/i686/nosegneg/libpthread.so.0
#2  0xb7417450 in _L_lock_677 () from /lib/i686/nosegneg/libpthread.so.0
#3  0xb7417321 in pthread_mutex_lock () from /lib/i686/nosegneg/libpthread.so.0
#4  0xb736aeb6 in pthread_mutex_lock () from /lib/i686/nosegneg/libc.so.6
#5  0xb68fedae in apc_pthreadmutex_lock () from /usr/lib/php/modules/apc.so
#6  0xb68f87a1 in apc_cache_user_delete () from /usr/lib/php/modules/apc.so
#7  0xb68f4ac4 in zif_apc_delete () from /usr/lib/php/modules/apc.so
#8  0xb7082169 in ?? () from /etc/httpd/modules/libphp5.so
#9  0xb7058b9e in execute () from /etc/httpd/modules/libphp5.so
#10 0xb7031216 in zend_execute_scripts () from /etc/httpd/modules/libphp5.so
#11 0xb6fd7f96 in php_execute_script () from /etc/httpd/modules/libphp5.so
#12 0xb70c06d3 in ?? () from /etc/httpd/modules/libphp5.so
#13 0xb7712561 in ap_run_handler ()
#14 0xb77162e6 in ap_invoke_handler ()
#15 0xb7722eb4 in ap_internal_redirect ()
#16 0xb71b0319 in ?? () from /etc/httpd/modules/mod_rewrite.so
#17 0xb7712561 in ap_run_handler ()
#18 0xb77162e6 in ap_invoke_handler ()
#19 0xb7723068 in ap_process_request ()
#20 0xb771fb58 in ?? ()
#21 0xb771add1 in ap_run_process_connection ()
#22 0xb77289aa in ?? ()
#23 0xb7728d3e in ?? ()
#24 0xb7729d13 in ap_mpm_run ()
#25 0xb76fcad2 in main ()

[Brynet-Inc]

Time to switch to nginx!

[chase]

I was looking at apache alternatives (mainly http://www.lighttpd.net/) but just replacing all the mod_rewrite rules looked like such a time consuming PITA that I just wanted to get the site running on the new server with a setup as similar to the old one as I could. I'll probably have to spend some more serious time looking into this in the future.

If baiduspider and bingbot would stop hammering the site it'd probably help too.

[gravaera]

Down with bing, don't let them index your site, stop them now...do it for software freedom huehuehue

[Brynet-Inc]

nginx's equivalent to mod_rewrite looks slightly more comprehensible, some relevant info:

http://wiki.nginx.org/NginxHttpRewriteModule
http://www.nginx.org/en/docs/http/conve ... rules.html

Also handy, http://wiki.nginx.org/ModuleComparisonMatrix.

[bubach]

Hmm, I could serve like 2 million hits per day on one of my sites without any problems on Apache, guess my custom PHP was a bit more tweaked towards performance than phpBB and mediawiki but still... Should never have to worry about any issues on a site this size, standard server settings suck.

[Jezze]

If you have performance problems you might look at adding a varnish layer.

https://www.varnish-cache.org/

Haven't tried it myself but it is suppose to be very good as a temporary fix

[brain]

If you have performance problems you might look at adding a varnish layer.

https://www.varnish-cache.org/

Haven't tried it myself but it is suppose to be very good as a temporary fix

I run a reasonably large and busy wiki too and ive found that using a reverse proxy does work wonders. We had all kinds of issues until we:

1) Installed reverse squid proxy
2) Moved seperated wiki and forums to two different virtul servers on the same box
3) Installed APC cache and memcached
4) Banned a lot of misbehaving spiders at the firewall and via useragent in apache, such as biadu and various spam harvesters