Unicorn 0.97.0 old master is never dying

Unicorn 0.97.0 old master is never dying

[ghazel]

I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
sending a USR2 to the 0.97.0 master launches a new master and all the
workers, but never kills the old master. The new workers retry binding
their admin TCP ports forever. Killing all the Unicorns and
downgrading to returns peace to the forest.

Here are the before_fork and after_fork from my config/unicorn.rb:
http://codepad.org/5LyVtyq7

Did something change with the Unicorn master in 0.97.0 that would
require a change in my config, or is this a bug?

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn 0.97.0 old master is never dying

[Eric Wong]

ghazel@gmail.com wrote:
> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
> sending a USR2 to the 0.97.0 master launches a new master and all the
> workers, but never kills the old master. The new workers retry binding
> their admin TCP ports forever. Killing all the Unicorns and
> downgrading to returns peace to the forest.
> 
> Here are the before_fork and after_fork from my config/unicorn.rb:
> http://codepad.org/5LyVtyq7
> 
> Did something change with the Unicorn master in 0.97.0 that would
> require a change in my config, or is this a bug?
> 
> -Greg

Hi Greg,

Odd, are you passing the pid file path via the command-line?

I don't see the pid directive in the config file you posted there, so
there was nothing to check with File.exists?.  Are you using `unicorn'
or `unicorn_rails'?

There were some startup changes in 0.97.0, but I couldn't reproduce what
you're seeing with 'pid'.  That said, I'm not sure if that method of
killing old masters in the after_fork is very common.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn 0.97.0 old master is never dying

[ghazel]

On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson@yhbt.net> wrote:
> ghazel@gmail.com wrote:
>> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
>> sending a USR2 to the 0.97.0 master launches a new master and all the
>> workers, but never kills the old master. The new workers retry binding
>> their admin TCP ports forever. Killing all the Unicorns and
>> downgrading to returns peace to the forest.
>>
>> Here are the before_fork and after_fork from my config/unicorn.rb:
>> http://codepad.org/5LyVtyq7
>>
>> Did something change with the Unicorn master in 0.97.0 that would
>> require a change in my config, or is this a bug?
>>
>> -Greg
>
> Hi Greg,
>
> Odd, are you passing the pid file path via the command-line?
>
> I don't see the pid directive in the config file you posted there, so
> there was nothing to check with File.exists?.  Are you using `unicorn'
> or `unicorn_rails'?
>
> There were some startup changes in 0.97.0, but I couldn't reproduce what
> you're seeing with 'pid'.  That said, I'm not sure if that method of
> killing old masters in the after_fork is very common.
>
> --
> Eric Wong
>

Eric,

I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is
the entire config file: http://codepad.org/v6lUsuzD

What is the preferred way to do an nginx-style restart? My system does
not really have the memory to start twice the number of workers.

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn 0.97.0 old master is never dying

[Eric Wong]

ghazel@gmail.com wrote:
> On Thu, Mar 18, 2010 at 1:40 AM, Eric Wong <normalperson@yhbt.net> wrote:
> > ghazel@gmail.com wrote:
> >> I upgraded to Unicorn 0.97.0 from 0.96.1. Unicorn starts fine, but
> >> sending a USR2 to the 0.97.0 master launches a new master and all the
> >> workers, but never kills the old master. The new workers retry binding
> >> their admin TCP ports forever. Killing all the Unicorns and
> >> downgrading to returns peace to the forest.
> >>
> >> Here are the before_fork and after_fork from my config/unicorn.rb:
> >> http://codepad.org/5LyVtyq7
> >>
> >> Did something change with the Unicorn master in 0.97.0 that would
> >> require a change in my config, or is this a bug?
> >>
> >> -Greg
> >
> > Hi Greg,
> >
> > Odd, are you passing the pid file path via the command-line?
> >
> > I don't see the pid directive in the config file you posted there, so
> > there was nothing to check with File.exists?.  Are you using `unicorn'
> > or `unicorn_rails'?
> >
> > There were some startup changes in 0.97.0, but I couldn't reproduce what
> > you're seeing with 'pid'.  That said, I'm not sure if that method of
> > killing old masters in the after_fork is very common.
> >
> > --
> > Eric Wong
> >
> 
> Eric,
> 
> I am using "unicorn_rails -D -E $ENV -c config/unicorn.rb". Here is
> the entire config file: http://codepad.org/v6lUsuzD

Ugh.  I can't even get to that file right now.  I actually prefer
to have config files inline so:

a) I can comment easily on them
b) I don't have to be online when reading them
   (I sync my mail to a local machine and lose connectivity often)


On a side note, maybe an strace/truss can help figure out what's
going on.  James seems to be having a similar problem in the other
thread...

> What is the preferred way to do an nginx-style restart? My system does
> not really have the memory to start twice the number of workers.

If you're doing it during low traffic periods, I would SIGTTOU away
most (maybe all but one) of your workers and then SIGHUP the master

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn 0.97.0 old master is never dying

[ghazel]

>> Here is the entire config file: http://codepad.org/v6lUsuzD
>
> Ugh.  I can't even get to that file right now.  I actually prefer
> to have config files inline so:
>
> a) I can comment easily on them
> b) I don't have to be online when reading them
>   (I sync my mail to a local machine and lose connectivity often)

We're still working on that internet that works when you're not
online. In the meantime:


# Sample configuration file for Unicorn (not Rack)
#
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete
# documentation.

require '/data/myapp/current/config/unicorn_constants'

# Use at least one worker per core if you're on a dedicated server,
# more will usually help for _short_ waits on databases/caches.
worker_processes UnicornConstants::NUM_WORKERS

# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
working_directory "/data/myapp/current" # available in 0.94.0+

# listen on both a Unix domain socket and a TCP port,
# we use a shorter backlog for quicker failover when busy
listen "/tmp/unicorn.sock", :backlog => 64
listen 8080, :tcp_nopush => true, :tries => -1

# nuke workers after X seconds instead of (60 seconds default)
timeout 300

# feel free to point this anywhere accessible on the filesystem
pid "/data/myapp/shared/pids/unicorn.pid"

# some applications/frameworks log to stderr or stdout, so prevent
# them from going to /dev/null when daemonized here:
stderr_path "/data/myapp/shared/log/unicorn.stderr.log"
stdout_path "/data/myapp/shared/log/unicorn.stdout.log"

# combine REE with "preload_app true" for memory savings
# http://rubyenterpriseedition.com/faq.html#adapt_apps_for_cow
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!

  # The following is only recommended for memory/DB-constrained
  # installations.  It is not needed if your system can house
  # twice as many worker_processes as you have configured.
  #
  # This allows a new master process to incrementally
  # phase out the old master process with SIGTTOU to avoid a
  # thundering herd (especially in the "preload_app false" case)
  # when doing a transparent upgrade.  The last worker spawned
  # will then kill off the old master process with a SIGQUIT.
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  # per-process listener ports for debugging/admin/migrations
  addr = "127.0.0.1:#{9000 + worker.nr}"
  server.listen(addr, :tries => -1, :delay => 5, :tcp_nopush => true)

  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection

  # if preload_app is true, then you may also want to check and
  # restart any other shared sockets/descriptors such as Memcached,
  # and Redis.  TokyoCabinet file handles are safe to reuse
  # between any number of forked children (assuming your kernel
  # correctly implements pread()/pwrite() system calls)
end



-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying