Unicorn completely ignores USR2 signal

Unicorn completely ignores USR2 signal

[Ehud Rosenberg]

Hi all,
I'm experiencing a rather strange problem with unicorn on my production server.
Although the config file states preload_app true, sending USR2 to the
master process does not generate any response, and it seems like
unicorn is ignoring the signal altogether.
On another server sending USR2 changes the master process to and (old)
state and starts a new master process successfully.
The problematic server is using RVM & bundler, so I'm assuming it's
somehow related (the other one is vanilla ruby).
Sending signals other than USR2 (QUIT, HUP) works just fine.
Is there a way to trace what's going on behind the scenes here?
Unicorn's log file is completely empty.
Thanks :)
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn completely ignores USR2 signal

[Eric Wong]

Ehud Rosenberg <ehudros@gmail.com> wrote:
> Hi all,
> I'm experiencing a rather strange problem with unicorn on my production server.
> Although the config file states preload_app true, sending USR2 to the
> master process does not generate any response, and it seems like
> unicorn is ignoring the signal altogether.
> On another server sending USR2 changes the master process to and (old)
> state and starts a new master process successfully.
> The problematic server is using RVM & bundler, so I'm assuming it's
> somehow related (the other one is vanilla ruby).

RVM could be a problem, especially if your ENV changed somehow and your
path to.  Can you dump out the START_CTX and ENV contents in the
before_exec hook?

  before_exec do |server|
    File.open("/tmp/start_ctx.dump", "ab") do |fp|
      PP.pp Unicorn::HttpServer::START_CTX, fp
      PP.pp ENV, fp
    end
  end

You may need to modify START_CTX if you're changing paths or if
somehow RVM gave you the wrong path to unicorn

START_CTX is documented here:
  http://unicorn.bogomips.org/Unicorn/HttpServer.html

> Sending signals other than USR2 (QUIT, HUP) works just fine.
> Is there a way to trace what's going on behind the scenes here?

Run strace (Linux) or equivalent on the master process.

> Unicorn's log file is completely empty.

This is the log you setup to point to stderr_path?  That shouldn't
happen...

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn completely ignores USR2 signal

[Eric Wong]

Any progress on this issue?  I'll have limited network access
the next few days...
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn completely ignores USR2 signal

[Alex Sharp]

Can you report the following? Ruby version (including the patch
number), OS and version? The linux kernel shipped with Ubuntu natty
11.04 has a known bug with signal trapping that affects ruby 1.9.2.
See here:

* https://github.com/bvandenbos/resque-scheduler/issues/90
* https://bugzilla.kernel.org/show_bug.cgi?id=32922

On Thu, Jul 7, 2011 at 4:18 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Any progress on this issue?  I'll have limited network access
> the next few days...
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>



-- 
Best,
alex sharp
github.com/ajsharp
twitter.com/ajsharp
alexjsharp.com
615.243.1472
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn completely ignores USR2 signal

[Ehud Rosenberg]

Eric Wong <normalperson <at> yhbt.net> writes:

> 
> Ehud Rosenberg <ehudros <at> gmail.com> wrote:
> > Hi all,
> > I'm experiencing a rather strange problem with unicorn on my production 
server.
> > Although the config file states preload_app true, sending USR2 to the
> > master process does not generate any response, and it seems like
> > unicorn is ignoring the signal altogether.
> > On another server sending USR2 changes the master process to and (old)
> > state and starts a new master process successfully.
> > The problematic server is using RVM & bundler, so I'm assuming it's
> > somehow related (the other one is vanilla ruby).
> 
> RVM could be a problem, especially if your ENV changed somehow and your
> path to.  Can you dump out the START_CTX and ENV contents in the
> before_exec hook?
> 
>   before_exec do |server|
>     File.open("/tmp/start_ctx.dump", "ab") do |fp|
>       PP.pp Unicorn::HttpServer::START_CTX, fp
>       PP.pp ENV, fp
>     end
>   end
> 
> You may need to modify START_CTX if you're changing paths or if
> somehow RVM gave you the wrong path to unicorn
> 
> START_CTX is documented here:
>   http://unicorn.bogomips.org/Unicorn/HttpServer.html
> 
> > Sending signals other than USR2 (QUIT, HUP) works just fine.
> > Is there a way to trace what's going on behind the scenes here?
> 
> Run strace (Linux) or equivalent on the master process.
> 
> > Unicorn's log file is completely empty.
> 
> This is the log you setup to point to stderr_path?  That shouldn't
> happen...
> 

Sorry for not getting back to you guys earlier. 
It seems like the entire setup is somehow broken, as the before_exec block does 
not seem to output anything to the logfiles/temp file (I know how weird that 
sounds). The stdout/stderr logs do not contain any info other than the regular 
"worker started" message.
I've tried 'puts' outside of the before_exec block and got the following output 
straight into my stdout (right after running sudo /etc/init.d/unicorn start):
{:argv=>["-D", "-E", "production", "-c", 
"/var/www/platform/current/config/unicorn.rb"], 0=>"/usr/local/rvm/gems/ruby-
1.9.2-head/bin/unicorn", :cwd=>"/var/www/platform/current"}

{"OLDPWD"=>"/var/www/platform/current", "SUDO_UID"=>"1000", "LOGNAME"=>"root", 
"USERNAME"=>"root", "PATH"=>"/usr/local/rvm/gems/ruby-1.9.2-
head/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6
/bin", "SHELL"=>"/bin/bash", "GEM_HOME"=>"/usr/local/rvm/gems/ruby-1.9.2-head",  
"PWD"=>"/var/www/platform/current", "RACK_ENV"=>"production", 
"BUNDLE_GEMFILE"=>"/var/www/platform/current/Gemfile", 
"BUNDLE_BIN_PATH"=>"/usr/local/rvm/gems/ruby-1.9.2-head/gems/bundler-
1.0.10/bin/bundle", "RUBYOPT"=>"-I/usr/local/rvm/gems/ruby-1.9.2-
head/gems/bundler-1.0.10/lib -rbundler/setup"}

The ENV output was truncated a little to remove irrelevant keys.

strace can be found here: 
https://gist.github.com/54cd784880b99a2b46d5/77718daec3c08227639a0619e02d387c4d0
cd9fd
I couldn't really figure out what's going on there - it complains the pid file 
can't be found although it's there, and it then writes its content to the oldbin 
file (which is never actually created in the pid directory). Permissions wise 
the directory is writable by pretty much anyone, and unicorn runs as root 
anyway.

The server is Ubuntu 10.04, ruby 1.9.2p53.
The server starts fine and the site is fully functional despite all the 
weirdness :)



_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying