unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
* unicorn doesn't restart properly after cap deploy (not using Bundler)
@ 2011-05-19  5:56 Jason Su
  2011-05-19  9:27 ` Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: Jason Su @ 2011-05-19  5:56 UTC (permalink / raw)
  To: mongrel-unicorn

Hey guys,

I'm sending USR2 to unicorn after cap deploy, and the old master is
getting replaced by a new master that doesn't work.

Here are some similar threads I found ...
http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html

I'm not using Bundler, and I have working_directory set in my unicorn config:

Unicorn::HttpServer::START_CTX[0] =
"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"
pid "/var/www/lookbook/shared/pids/unicorn.pid"
stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
  GC.copy_on_write_friendly = true
end
before_fork do |server, worker|
  STDERR.puts "BEFORE FORK:"
  STDERR.puts ENV.inspect
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
  if old_pid != server.pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
    end
  end

end
after_fork do |server, worker|
  STDERR.puts "AFTER FORK:"
  STDERR.puts ENV.inspect

  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
  begin
    uid, gid = Process.euid, Process.egid
    user, group = 'apache', 'apache'
    target_uid = Etc.getpwnam(user).uid
    target_gid = Etc.getgrnam(group).gid
    worker.tmp.chown(target_uid, target_gid)
    if uid != target_uid || gid != target_gid
      Process.initgroups(user, target_gid)
      Process::GID.change_privilege(target_gid)
      Process::UID.change_privilege(target_uid)
    end
  rescue => e
    if RAILS_ENV == 'development'
      STDERR.puts "couldn't change user, oh well"
    else
      raise e
    end
  end
end
before_exec do |server|
  STDERR.puts "BEFORE EXEC:"
  ENV["PATH"] =
"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin"
  STDERR.puts ENV.inspect
end



Below is the output from unicorn.stderr.log.. I'm not sure if
UNICORN_FD has something to do with it?
1) starting unicorn
BEFORE FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production"}
AFTER FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production"}


2) restarting unicorn (kill -USR2)

BEFORE EXEC:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}
I, [2011-05-18T18:13:56.318531 #510]  INFO -- : inherited
addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
I, [2011-05-18T18:13:56.318819 #510]  INFO -- : Refreshing Gem list
BEFORE FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}
AFTER FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}

3) cap deploy with symlinks + restart unicorn (kill -USR2)

BEFORE EXEC:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}
I, [2011-05-18T18:17:13.732600 #1589]  INFO -- : inherited
addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
I, [2011-05-18T18:17:13.732942 #1589]  INFO -- : Refreshing Gem list
BEFORE FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}
AFTER FORK:
{"SHELL"=>"/bin/bash", "TERM"=>"xterm", "SSH_CLIENT"=>"24.23.182.32
38170 22", "OLDPWD"=>"/root", "SSH_TTY"=>"/dev/pts/5", "USER"=>"root",
"SSH_AUTH_SOCK"=>"/tmp/ssh-eeoDcvH461/agent.461",
"PATH"=>"/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local:/opt/ruby-enterprise-1.8.7-2010.02/bin",
"MAIL"=>"/var/mail/root", "PWD"=>"/var/www/lookbook/current",
"HOME"=>"/root", "SHLVL"=>"2", "LOGNAME"=>"root",
"SSH_CONNECTION"=>"24.23.182.32 38170 98.158.22.100 22",
"_"=>"/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails",
"RAILS_ENV"=>"production", "UNICORN_FD"=>"3"}
worker=1 ready


any help appreciated, thanks :D
Jason
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: unicorn doesn't restart properly after cap deploy (not using Bundler)
  2011-05-19  5:56 unicorn doesn't restart properly after cap deploy (not using Bundler) Jason Su
@ 2011-05-19  9:27 ` Eric Wong
  2011-05-19 20:26   ` Jason Su
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2011-05-19  9:27 UTC (permalink / raw)
  To: unicorn list

Jason Su <jason@lookbook.nu> wrote:
> Hey guys,
> 
> I'm sending USR2 to unicorn after cap deploy, and the old master is
> getting replaced by a new master that doesn't work.

Can you expand on "doesn't work"?

> Here are some similar threads I found ...
> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html
> 
> I'm not using Bundler, and I have working_directory set in my unicorn config:

I don't see where you have working_directory set below

> Unicorn::HttpServer::START_CTX[0] =
> "/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"

You shouldn't need to touch START_CTX unless you're switching Ruby
installations.  It's rope to hang yourself with.

I would simplify the config file as much as possible if you're
debugging a problem.

> pid "/var/www/lookbook/shared/pids/unicorn.pid"
> stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
> stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
> preload_app true
> if GC.respond_to?(:copy_on_write_friendly=)
>   GC.copy_on_write_friendly = true
> end
> before_fork do |server, worker|
>   STDERR.puts "BEFORE FORK:"
>   STDERR.puts ENV.inspect
>   defined?(ActiveRecord::Base) and
>     ActiveRecord::Base.connection.disconnect!

The auto-killing old workers bit is fragile and you shouldn't need
it unless you're very low on memory.

>   if old_pid != server.pid
>     begin
>       sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
>       Process.kill(sig, File.read(old_pid).to_i)
>     rescue Errno::ENOENT, Errno::ESRCH
>     end
>   end
> 
> end
> after_fork do |server, worker|
>   STDERR.puts "AFTER FORK:"
>   STDERR.puts ENV.inspect
> 
>   defined?(ActiveRecord::Base) and
>     ActiveRecord::Base.establish_connection
>   begin

If you want to use user switching, the "user" directive which
appeared a while back is much simpler and less error prone.

>     uid, gid = Process.euid, Process.egid
>     user, group = 'apache', 'apache'
>     target_uid = Etc.getpwnam(user).uid
>     target_gid = Etc.getgrnam(group).gid
>     worker.tmp.chown(target_uid, target_gid)
>     if uid != target_uid || gid != target_gid
>       Process.initgroups(user, target_gid)
>       Process::GID.change_privilege(target_gid)
>       Process::UID.change_privilege(target_uid)
>     end

<snip>

> Below is the output from unicorn.stderr.log.. I'm not sure if
> UNICORN_FD has something to do with it?

UNICORN_FD is absolutely for the old master tells the new master about
the listener sockets to inherit.  Don't touch it.

<snip>

> 3) cap deploy with symlinks + restart unicorn (kill -USR2)

<snip>

> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3

You should make sure your socket is in a shared path, not current.

Whatever's in current gets replaced by Capistrano, right?  I'm not sure
if tmp is a symlink that's relinked on a new deploy, but there could be
a race condition where the path to the socket is unreachable and
nginx can't see it.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: unicorn doesn't restart properly after cap deploy (not using Bundler)
  2011-05-19  9:27 ` Eric Wong
@ 2011-05-19 20:26   ` Jason Su
  0 siblings, 0 replies; 3+ messages in thread
From: Jason Su @ 2011-05-19 20:26 UTC (permalink / raw)
  To: unicorn list

You were right, the socket was not in a shared path... D'oh!

Thanks so much~

On Thu, May 19, 2011 at 2:27 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Jason Su <jason@lookbook.nu> wrote:
>> Hey guys,
>>
>> I'm sending USR2 to unicorn after cap deploy, and the old master is
>> getting replaced by a new master that doesn't work.
>
> Can you expand on "doesn't work"?
>
>> Here are some similar threads I found ...
>> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000733.html
>> http://rubyforge.org/pipermail/mongrel-unicorn/2010-October/000717.html
>>
>> I'm not using Bundler, and I have working_directory set in my unicorn config:
>
> I don't see where you have working_directory set below
>
>> Unicorn::HttpServer::START_CTX[0] =
>> "/opt/ruby-enterprise-1.8.7-2010.02/bin/unicorn_rails"
>
> You shouldn't need to touch START_CTX unless you're switching Ruby
> installations.  It's rope to hang yourself with.
>
> I would simplify the config file as much as possible if you're
> debugging a problem.
>
>> pid "/var/www/lookbook/shared/pids/unicorn.pid"
>> stderr_path "/var/www/lookbook/shared/log/unicorn.stderr.log"
>> stdout_path "/var/www/lookbook/shared/log/unicorn.stdout.log"
>> preload_app true
>> if GC.respond_to?(:copy_on_write_friendly=)
>>   GC.copy_on_write_friendly = true
>> end
>> before_fork do |server, worker|
>>   STDERR.puts "BEFORE FORK:"
>>   STDERR.puts ENV.inspect
>>   defined?(ActiveRecord::Base) and
>>     ActiveRecord::Base.connection.disconnect!
>
> The auto-killing old workers bit is fragile and you shouldn't need
> it unless you're very low on memory.
>
>>   if old_pid != server.pid
>>     begin
>>       sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
>>       Process.kill(sig, File.read(old_pid).to_i)
>>     rescue Errno::ENOENT, Errno::ESRCH
>>     end
>>   end
>>
>> end
>> after_fork do |server, worker|
>>   STDERR.puts "AFTER FORK:"
>>   STDERR.puts ENV.inspect
>>
>>   defined?(ActiveRecord::Base) and
>>     ActiveRecord::Base.establish_connection
>>   begin
>
> If you want to use user switching, the "user" directive which
> appeared a while back is much simpler and less error prone.
>
>>     uid, gid = Process.euid, Process.egid
>>     user, group = 'apache', 'apache'
>>     target_uid = Etc.getpwnam(user).uid
>>     target_gid = Etc.getgrnam(group).gid
>>     worker.tmp.chown(target_uid, target_gid)
>>     if uid != target_uid || gid != target_gid
>>       Process.initgroups(user, target_gid)
>>       Process::GID.change_privilege(target_gid)
>>       Process::UID.change_privilege(target_uid)
>>     end
>
> <snip>
>
>> Below is the output from unicorn.stderr.log.. I'm not sure if
>> UNICORN_FD has something to do with it?
>
> UNICORN_FD is absolutely for the old master tells the new master about
> the listener sockets to inherit.  Don't touch it.
>
> <snip>
>
>> 3) cap deploy with symlinks + restart unicorn (kill -USR2)
>
> <snip>
>
>> addr=/var/www/lookbook/current/tmp/sockets/unicorn.sock fd=3
>
> You should make sure your socket is in a shared path, not current.
>
> Whatever's in current gets replaced by Capistrano, right?  I'm not sure
> if tmp is a symlink that's relinked on a new deploy, but there could be
> a race condition where the path to the socket is unreachable and
> nginx can't see it.
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2011-05-19 20:46 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-05-19  5:56 unicorn doesn't restart properly after cap deploy (not using Bundler) Jason Su
2011-05-19  9:27 ` Eric Wong
2011-05-19 20:26   ` Jason Su

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).