Re: funky process tree + stillborn masters

[Eric Wong <normalperson@yhbt.net>]

Jamie Wilkinson <jamie@tramchase.com> wrote:
> I've also produced straces of the *original* master during USR2
> restarts, both a success trace and a failure trace.  Here's a tarball
> with both complete traces as well as filtered/grepp'd ones:
> http://jamiedubs.com/files/unicorn-strace.tgz
> 
> I've also found that kill -9'ing the 1st worker of the new orphaned
> master allows it to continue operation as normal (spinning up workers
> and taking control from the original master) -- suggesting something
> is up with just that first worker (!). I'm going to keep noodling with
> before_/after_fork strategies.

Hi Jamie, your successful straces have this oddity in them:

  poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 1 ([{fd=7, revents=POLLIN|POLLHUP}
  read(7, ""..., 8192)                    = 0
  shutdown(7, 2 /* send and receive */)   = -1 ENOTCONN (Transport endpoint is not
  close(7)                                = 0

> Eric Wong <normalperson@yhbt.net> wrote:
> > Anything in your before_fork/after_fork hooks?  Since it looks like
> > you're on a Linux system, can you strace the master while you send
> > it a USR2 and see if anything strange happens?
> 
> The only real contents of our before_hook is a
> send-QUIT-on-first-worker, which I swapped out for the default SIGTTOU
> behavior. No change.

It looks like you should be disconnecting/reconnecting whatever is using
fd=7 in the before_fork/after_fork hooks respectively.

Unicorn (and mainline Ruby) do not use the poll() system call (which is
why that strace raised red flags), so there's another C extension
calling poll() in your master process (not good).  I remember one (or
all of) the Ruby Postgres libraries using poll() internally, and there
are likely other things that use poll() in Ruby as well.

Perhaps something like this in the config (assuming you're using
Rails ActiveRecord) (also see http://unicorn.bogomips.org/examples/)

  before_fork do |server,worker|
    defined?(ActiveRecord::Base) and
      ActiveRecord::Base.connection.disconnect!
  end

  after_fork do |server,worker|
    defined?(ActiveRecord::Base) and
      ActiveRecord::Base.establish_connection
  end

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying