funky process tree + stillborn masters

[Jamie Wilkinson <jamie@tramchase.com>]

Since upgrading bundler (but applying the RUBYOPT path fixes) we've started experiencing intermittent, difficult-to-isolate USR2 restart failures.

After a USR2 signal our process tree winds up looking like this, with several master-esque processes listed as children (but without the "worker[N]" label):

app      14402  4.4  0.8 199612 70264 ?        S    14:07   0:04 unicorn_rails master -c config/unicorn.rb -E production -D
app      14433  0.0  0.8 204540 68504 ?        Sl   14:07   0:00  \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D
app      14435  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D
app      14438  0.0  0.8 199748 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
app      14440  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D
app      14442  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D
app      14445  0.0  0.8 199760 65840 ?        S    14:07   0:00  \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D
app      14447  0.0  0.8 204540 68508 ?        Sl   14:07   0:00  \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D
app      14449  0.0  0.8 204780 69272 ?        Sl   14:07   0:00  \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D

Sending another USR2 signal will bring a new master into the mix as a child, spins up a single child worker of its own (which also resembles the "/usr/bin/ruby1.8" master-esque processes), and then fails to continue. 

Further USR2 restarts will obviously do nothing, and we're forced to either kill -9 the stillborn master or cold-restart all of the unicorns. Nothing out of the ordinary is dumped to stderr or stdout

Starting unicorns fresh produces a nice process list where every child is listed cleanly as "unicorn_rails worker[N]"

We only have this issue in one of our applications, on a machine that has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've also only noticed this since upgrading to bundler from bundler08

Are the goofy worker processes in the process tree a real problem, or just a red herring?

-jamie
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying