Issue with unicorn not starting via bluepill?

Issue with unicorn not starting via bluepill?

[James Cox]

trying to start my unicorn via bluepill, and running into this :

Starting unicorn...\n{:unicorn_options=>
  {:config_file=>"/home/configs/unicorn/marketing.rb", :listeners=>[]},
 :daemonize=>true,
 :app=>
  #<Proc:0x000000001042c718@/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/bin/unicorn_rails:120>}
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/cache
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/pids
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/sessions
Exception `Errno::EEXIST' at
/usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
- File exists - tmp/sockets
Exception `EOFError' at
/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45
- end of file reached
master failed to start, check stderr log for details


it's a rather complicated setup etc, but essentially i've got bluepill
instructing a start script which sets the right version of ruby just
before hand (i.e. the rvm stuff).

I can run it manually by going into the RAILS_ROOT and running it, but
trying to run via bluepill explodes.

any ideas?

Thanks,
James
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Issue with unicorn not starting via bluepill?

[Eric Wong]

James Cox <james@imaj.es> wrote:
> trying to start my unicorn via bluepill, and running into this :
> 
> Starting unicorn...\n{:unicorn_options=>
>   {:config_file=>"/home/configs/unicorn/marketing.rb", :listeners=>[]},
>  :daemonize=>true,
>  :app=>
>   #<Proc:0x000000001042c718@/usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/bin/unicorn_rails:120>}
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/cache
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/pids
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/sessions
> Exception `Errno::EEXIST' at
> /usr/local/rvm/rubies/ree-1.8.7-2010.01/lib/ruby/1.8/fileutils.rb:243
> - File exists - tmp/sockets
> Exception `EOFError' at
> /usr/local/rvm/gems/ree-1.8.7-2010.01/gems/unicorn-0.97.0/lib/unicorn/launcher.rb:45
> - end of file reached
> master failed to start, check stderr log for details
> 
> 
> it's a rather complicated setup etc, but essentially i've got bluepill
> instructing a start script which sets the right version of ruby just
> before hand (i.e. the rvm stuff).
> 
> I can run it manually by going into the RAILS_ROOT and running it, but
> trying to run via bluepill explodes.
> 
> any ideas?

I'm not familiar with Bluepill, it's a process monitor of some
sort.  Both you and Greg are using unicorn_rails which did get
some overhauling in Unicorn 0.97.0.  Everything seems fine on
my end, but maybe there's a piece of the puzzle I'm overlooking..

I'll dig further.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Issue with unicorn not starting via bluepill?

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> James Cox <james@imaj.es> wrote:
> > trying to start my unicorn via bluepill, and running into this :

> > master failed to start, check stderr log for details

Both of you:

Was there anything useful in stderr_path?  The daemomized
process can never output to the terminal, so it had to
log somewhere... There's also strace, but looking at the
Unicorn code I haven't been able to find much (I'm very
sleep deprived right now, though).

@Greg: I got through to /v6lUsuzD finally, can you put some
debug statements around your before_fork hook and print out
the pid path name and whether it exists or not?

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Issue with unicorn not starting via bluepill?

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
> > James Cox <james@imaj.es> wrote:
> > > trying to start my unicorn via bluepill, and running into this :
> 
> > > master failed to start, check stderr log for details
> 
> Both of you:

Hi, did you guys ever figure this out?

> Was there anything useful in stderr_path?  The daemomized
> process can never output to the terminal, so it had to
> log somewhere... There's also strace, but looking at the
> Unicorn code I haven't been able to find much (I'm very
> sleep deprived right now, though).
> 
> @Greg: I got through to /v6lUsuzD finally, can you put some
> debug statements around your before_fork hook and print out
> the pid path name and whether it exists or not?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Issue with unicorn not starting via bluepill?

[James Cox]

Not yet- i've been fixing other things first. Looking at this again shortly

On Tue, Mar 23, 2010 at 2:54 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
>> Eric Wong <normalperson@yhbt.net> wrote:
>> > James Cox <james@imaj.es> wrote:
>> > > trying to start my unicorn via bluepill, and running into this :
>>
>> > > master failed to start, check stderr log for details
>>
>> Both of you:
>
> Hi, did you guys ever figure this out?
>
>> Was there anything useful in stderr_path?  The daemomized
>> process can never output to the terminal, so it had to
>> log somewhere... There's also strace, but looking at the
>> Unicorn code I haven't been able to find much (I'm very
>> sleep deprived right now, though).
>>
>> @Greg: I got through to /v6lUsuzD finally, can you put some
>> debug statements around your before_fork hook and print out
>> the pid path name and whether it exists or not?
>



-- 
James Cox,
Consultant, Raconteur, Photographer, Entrepreneur
t: 07968 349990  e: james@imaj.es w: http://imaj.es/
talk: http://twitter.com/imajes photos: http://flickr.com/imajes
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Issue with unicorn not starting via bluepill?

[ghazel]

On Fri, Mar 19, 2010 at 1:57 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Eric Wong <normalperson@yhbt.net> wrote:
>> James Cox <james@imaj.es> wrote:
>> > trying to start my unicorn via bluepill, and running into this :
>
>> > master failed to start, check stderr log for details

For reference, this is not the error I get. I'm running "upgrade",
which is sig USR2, and getting two masters (one marked (old)) and two
sets of children.

> Both of you:
>
> Was there anything useful in stderr_path?  The daemomized
> process can never output to the terminal, so it had to
> log somewhere... There's also strace, but looking at the
> Unicorn code I haven't been able to find much (I'm very
> sleep deprived right now, though).

The only thing out of the ordinary I see in the log is:

retrying in 5 seconds (infinite tries left)
adding listener failed addr=127.0.0.1:9011 (in use)

as each of the admin interface TCP socket listeners fails on the
workers because there are still the old workers running.


> @Greg: I got through to /v6lUsuzD finally, can you put some
> debug statements around your before_fork hook and print out
> the pid path name and whether it exists or not?

server.config[:pid]: "/data/myapp/shared/pids/unicorn.pid"
exists?: true
old_pid: "/data/myapp/shared/pids/unicorn.pid.oldbin"
exists?: true
server.pid: "/data/myapp/shared/pids/unicorn.pid"
old_pid: "/data/myapp/shared/pids/unicorn.pid.oldbin"
server.pid != old_pid: true

It looks like trying to send a SIGQUIT to the old master does nothing.
Sending it a SIGTTOU does seem to reduce the number of workers, but
even when there are 0 left it does not die after a SIGQUIT.

Looking at the Unicorn source and my config:
http://codepad.org/v6lUsuzD I'm not sure what from the 0.96.1 was
supposed to be sending SIGTTOU. There are not any being sent, from
what I can tell. In the 0.96.1 to 0.96.1 upgrade case, I see the old
master get a bunch of SIGQUITs and reap its own children. In the
0.99.0 case I see no SIGQUITs arrive at the old master... In fact,
just running 0.99.0 and trying to send it a SIGQUIT does nothing.
SIGINT works fine, so do TTOU, TTIN, USR1 etc. I added a log message
to the trap handler registered in trap_deferred, and it is never being
called...

Turns out, I had query_trace installed, and the change in to trap
handler initialization registered it before query_trace initialized.
Later, query_trace registered a QUIT handler and did not call any
existing handlers. I have since uninstalled that broken junk.

-Greg
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying