* "unicorn -D" always returns 0 "success" (even when failed to load)
@ 2009-12-23 2:20 Iñaki Baz Castillo
2009-12-23 7:26 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-23 2:20 UTC (permalink / raw)
To: mongrel-unicorn
Hi, I'm writing a Debian init script for unicorn and realized that when
starting unicorn with daemonize option, the command always returns 0, even if
the start action failed (due for example Errno::EADDRINUSE).
Returning 0 in such case is not good as it breaks service init scripts or
service controllers (as HeartBeat) that fully rely on the appropriate exit
code.
Is there some way to determine if unicorn failed to start when using "-D"?
Another related issue: When the Rack config.ru file contains some error (as a
typo) the worker(s) returns 1 (at the moment usually). Then unicorn master
process reapes the terminated worker process and restarts it. Of course it
would fail again and again. Anyhow "unicorn -D" returns 0 again (success).
Usually if a worker (all the workers) fail to start at the moment of running
it, it obviously means that there is some error in the application with
prevents it to run. It could be great if Unicorn could detect it.
For that I suggest something as a new option "--validation-time TIME". Let's
suppose TIME is 5 seconds. In case *all* the workers fail within 5 seconds
after starting unicorn, then unicorn understands that the Rack application is
wrong (or any other error as Errno::EADDRINUSE) so terminates all the workers
and itself (and hopefully returns 1 or any other non-zero exit status).
Of course, all the above means that Unicorn should wait TIME seconds before
being daemonized (so after TIME seconds it can decide which code to return).
Does it make sense? Thanks a lot.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-23 2:20 Iñaki Baz Castillo
@ 2009-12-23 7:26 ` Eric Wong
2009-12-23 10:22 ` Iñaki Baz Castillo
2009-12-25 23:58 ` Iñaki Baz Castillo
0 siblings, 2 replies; 25+ messages in thread
From: Eric Wong @ 2009-12-23 7:26 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I'm writing a Debian init script for unicorn and realized that when
> starting unicorn with daemonize option, the command always returns 0, even if
> the start action failed (due for example Errno::EADDRINUSE).
>
> Returning 0 in such case is not good as it breaks service init scripts or
> service controllers (as HeartBeat) that fully rely on the appropriate exit
> code.
>
> Is there some way to determine if unicorn failed to start when using "-D"?
Hi Iñaki,
No way to determine that currently, as I've never encountered the need.
For validating startups, most folks I know have specific endpoint(s) in
application dedicated to checks. That way they can get way more info
and all the way down into stack including things like database/memcached
connections.
Anything less is superficial because they can fail to detect other
misconfigurations (including stuff like wrong RAILS_ENV); not just
startup errors.
> Another related issue: When the Rack config.ru file contains some error (as a
> typo) the worker(s) returns 1 (at the moment usually). Then unicorn master
> process reapes the terminated worker process and restarts it. Of course it
> would fail again and again. Anyhow "unicorn -D" returns 0 again (success).
>
> Usually if a worker (all the workers) fail to start at the moment of running
> it, it obviously means that there is some error in the application with
> prevents it to run. It could be great if Unicorn could detect it.
I'm generally hesitant to introduce code/features/bloat that aren't
helpful to people who properly test configurations before deploying :)
Adding this code doesn't solve problems. Either way the site can become
inaccessible/unusable and problems are noticeable very quickly.
If there's sufficient interest, I'll consider accepting a small patch
for this. Right now I'm unconvinced...
> For that I suggest something as a new option "--validation-time TIME". Let's
> suppose TIME is 5 seconds. In case *all* the workers fail within 5 seconds
> after starting unicorn, then unicorn understands that the Rack application is
> wrong (or any other error as Errno::EADDRINUSE) so terminates all the workers
> and itself (and hopefully returns 1 or any other non-zero exit status).
>
> Of course, all the above means that Unicorn should wait TIME seconds before
> being daemonized (so after TIME seconds it can decide which code to return).
>
> Does it make sense? Thanks a lot.
We avoid introducing command-line options since they're unlikely to be
compatible with "rackup" parsing of the "#\" lines in .ru files.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-23 7:26 ` Eric Wong
@ 2009-12-23 10:22 ` Iñaki Baz Castillo
2009-12-23 20:35 ` Eric Wong
2009-12-25 23:58 ` Iñaki Baz Castillo
1 sibling, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-23 10:22 UTC (permalink / raw)
To: mongrel-unicorn
El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> No way to determine that currently, as I've never encountered the need.
>
> For validating startups, most folks I know have specific endpoint(s) in
> application dedicated to checks. That way they can get way more info
> and all the way down into stack including things like database/memcached
> connections.
Yes, you are right and I agree now. I was thinking in Unicorn as a final
application rather than a HTTP applications container. It's like Apache, no
body expects Apache to return 1 and exit in case a PHP application has a typo
:)
> > Usually if a worker (all the workers) fail to start at the moment of
> > running it, it obviously means that there is some error in the
> > application with prevents it to run. It could be great if Unicorn could
> > detect it.
>
> I'm generally hesitant to introduce code/features/bloat that aren't
> helpful to people who properly test configurations before deploying :)
>
> Adding this code doesn't solve problems. Either way the site can become
> inaccessible/unusable and problems are noticeable very quickly.
Sure, but how to explain that to Hearteat? :)
> If there's sufficient interest, I'll consider accepting a small patch
> for this. Right now I'm unconvinced...
I've taken a look to the code and such a feature would definitively make it
ugly (IMHO).
So I'm thinking in a different approach: a custom script to check the status
of the application. Such script would communicate with the application (maybe
using DRB). If the DRB connection fails (because the workers fail to start)
then it means that something wrong occurs. And such script would also return
the whole status (DB connections and so).
I would include such script as "status" option in a service init script. The
"start" action would call "status" after N seconds and decide if the
*application* is running or not (if not then kill unicorn and exit 1).
PS: Since Unicorn makes usage of signals, is there any way to determine (by
using a signal) if the process running with some PID is an unicorn process?
This is: usually to verify the process status the PID file/value is inspected.
If there is a process running under such PID then we "assume" that process is
Unicorn. In case that PID is stolen by any other process (since PID file
wasn't deleted) nobody realizes of it.
A way to determine it could be asking to Unicorn master process (using i.e.
DRB) its PID and match it against the PID value stored in the pidfile. Of
course it would require some kind of communication with unicorn master process
to get its PID, is it possible now in some way?
> > For that I suggest something as a new option "--validation-time TIME".
> > Let's suppose TIME is 5 seconds. In case *all* the workers fail within 5
> > seconds after starting unicorn, then unicorn understands that the Rack
> > application is wrong (or any other error as Errno::EADDRINUSE) so
> > terminates all the workers and itself (and hopefully returns 1 or any
> > other non-zero exit status).
> >
> > Of course, all the above means that Unicorn should wait TIME seconds
> > before being daemonized (so after TIME seconds it can decide which code
> > to return).
> >
> > Does it make sense? Thanks a lot.
>
> We avoid introducing command-line options since they're unlikely to be
> compatible with "rackup" parsing of the "#\" lines in .ru files.
I didn't know about such options in .ru file. Are those options supposed to be
read by the backend?
Thanks a lot.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-23 10:22 ` Iñaki Baz Castillo
@ 2009-12-23 20:35 ` Eric Wong
2009-12-24 9:30 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-23 20:35 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> > > Usually if a worker (all the workers) fail to start at the moment of
> > > running it, it obviously means that there is some error in the
> > > application with prevents it to run. It could be great if Unicorn could
> > > detect it.
> >
> > I'm generally hesitant to introduce code/features/bloat that aren't
> > helpful to people who properly test configurations before deploying :)
> >
> > Adding this code doesn't solve problems. Either way the site can become
> > inaccessible/unusable and problems are noticeable very quickly.
>
> Sure, but how to explain that to Hearteat? :)
I had a lot of trouble coming to terms with this one myself :)
Eventually I reasoned that the heartbeat mechanism is to protect against
unforeseen failures way down the line (and sometimes outside of the
application's control).
It's (hopefully) common and expected to check applications shortly
following deployments, so heartbeat isn't for that.
However applications have a tendency to break in unforseen ways, and
only in production, and at the worst times. These failures happen well
after the app is deployed :)
In my experience, it's often the least used/trafficked parts of the
application that fail, too.
Of course if you write perfect software that handles every possible
failure case and don't need heartbeat (nor multiple processes because
your code is super-efficient :), check out Zbatery[1]
[1] http://zbatery.bogomip.org/
> > If there's sufficient interest, I'll consider accepting a small patch
> > for this. Right now I'm unconvinced...
>
> I've taken a look to the code and such a feature would definitively make it
> ugly (IMHO).
>
> So I'm thinking in a different approach: a custom script to check the status
> of the application. Such script would communicate with the application (maybe
> using DRB). If the DRB connection fails (because the workers fail to start)
> then it means that something wrong occurs. And such script would also return
> the whole status (DB connections and so).
> I would include such script as "status" option in a service init script. The
> "start" action would call "status" after N seconds and decide if the
> *application* is running or not (if not then kill unicorn and exit 1).
>
> PS: Since Unicorn makes usage of signals, is there any way to determine (by
> using a signal) if the process running with some PID is an unicorn process?
>
> This is: usually to verify the process status the PID file/value is inspected.
> If there is a process running under such PID then we "assume" that process is
> Unicorn. In case that PID is stolen by any other process (since PID file
> wasn't deleted) nobody realizes of it.
Under modern Linux systems, you have several options...
tr '\0' ' ' < /proc/$PID/cmdline # show the command-line
pmap $PID | grep unicorn_http.so # not many other things load this
lsof -p $PID # can check listening ports/log files
Unfortunately I don't use other OSes enough to remember the
equivalents...
> A way to determine it could be asking to Unicorn master process (using i.e.
> DRB) its PID and match it against the PID value stored in the pidfile. Of
> course it would require some kind of communication with unicorn master process
> to get its PID, is it possible now in some way?
I don't see how being under a specific PID actually matters. Why not
just write a script that sends an HTTP request to check?
Workers will die soon after the first request, or (timeout/2) seconds if
a master dies anyways.
> > We avoid introducing command-line options since they're unlikely to be
> > compatible with "rackup" parsing of the "#\" lines in .ru files.
>
> I didn't know about such options in .ru file. Are those options
> supposed to be read by the backend?
I don't believe it's well-documented, I only found out from reading the
rackup code and I'm not really a fan of magic comments, but they're only
for the basic command-line arguments rackup handles:
$ rackup -h
Usage: rackup [ruby options] [rack options] [rackup config]
Ruby options:
-e, --eval LINE evaluate a LINE of code
-d, --debug set debugging flags (set $DEBUG to true)
-w, --warn turn warnings on for your script
-I, --include PATH specify $LOAD_PATH (may be used more than once)
-r, --require LIBRARY require the library, before executing your script
Rack options:
-s, --server SERVER serve using SERVER (webrick/mongrel)
-o, --host HOST listen on HOST (default: 0.0.0.0)
-p, --port PORT use PORT (default: 9292)
-E, --env ENVIRONMENT use ENVIRONMENT for defaults (default: development)
-D, --daemonize run daemonized in the background
-P, --pid FILE file to store PID (default: rack.pid)
Common options:
-h, --help Show this message
--version Show version
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-23 20:35 ` Eric Wong
@ 2009-12-24 9:30 ` Iñaki Baz Castillo
2009-12-24 19:34 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-24 9:30 UTC (permalink / raw)
To: mongrel-unicorn
El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> > > Adding this code doesn't solve problems. Either way the site can
> > > become inaccessible/unusable and problems are noticeable very quickly.
> >
> > Sure, but how to explain that to Hearteat? :)
>
> I had a lot of trouble coming to terms with this one myself :)
>
> Eventually I reasoned that the heartbeat mechanism is to protect against
> unforeseen failures way down the line (and sometimes outside of the
> application's control).
>
> It's (hopefully) common and expected to check applications shortly
> following deployments, so heartbeat isn't for that.
Yes, you are right.
> However applications have a tendency to break in unforseen ways, and
> only in production, and at the worst times. These failures happen well
> after the app is deployed :)
Yes, every application breaks just in the moment that it shouldn't :)
However I'm building my app now and when start it it usually has errors, and
that's good as it means I'm working :)
But sincerelly I would like to see the error just once rather than many times
due to workers respawn :)
Also, is there any "interval" parameter so a worker is not re-spawned until
such timeout (after die).
> > This is: usually to verify the process status the PID file/value is
> > inspected. If there is a process running under such PID then we "assume"
> > that process is Unicorn. In case that PID is stolen by any other process
> > (since PID file wasn't deleted) nobody realizes of it.
>
> Under modern Linux systems, you have several options...
>
> tr '\0' ' ' < /proc/$PID/cmdline # show the command-line
>
> pmap $PID | grep unicorn_http.so # not many other things load this
>
> lsof -p $PID # can check listening ports/log files
Yes! this is much better! :)
> Unfortunately I don't use other OSes enough to remember the
> equivalents...
The server will run just on Linux (Unix) server :)
> > A way to determine it could be asking to Unicorn master process (using
> > i.e. DRB) its PID and match it against the PID value stored in the
> > pidfile. Of course it would require some kind of communication with
> > unicorn master process to get its PID, is it possible now in some way?
>
> I don't see how being under a specific PID actually matters. Why not
> just write a script that sends an HTTP request to check?
>
> Workers will die soon after the first request, or (timeout/2) seconds if
> a master dies anyways.
Well, in fact I want to code some console utility to get information from the
application (loaded configuration, check the database(s) connection and so).
For that I was thinking in a DRb server running in the master because it must
bind in some TCP port.
The problem is that I should stop the DRb server before master reexec to avoid
EADDINUSE. Well, I could code some rescue block...
> > I didn't know about such options in .ru file. Are those options
> > supposed to be read by the backend?
>
> I don't believe it's well-documented, I only found out from reading the
> rackup code and I'm not really a fan of magic comments, but they're only
> for the basic command-line arguments rackup handles:
>
> $ rackup -h
...
> -D, --daemonize run daemonized in the background
Why "daemonize" is not present in unicornf configuration file?
Really thanks a lot for all your help.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-24 9:30 ` Iñaki Baz Castillo
@ 2009-12-24 19:34 ` Eric Wong
2009-12-25 22:09 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-24 19:34 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> However I'm building my app now and when start it it usually has errors, and
> that's good as it means I'm working :)
> But sincerelly I would like to see the error just once rather than many times
> due to workers respawn :)
>
> Also, is there any "interval" parameter so a worker is not re-spawned until
> such timeout (after die).
You can sleep in the before_fork hook.
> Well, in fact I want to code some console utility to get information from the
> application (loaded configuration, check the database(s) connection and so).
> For that I was thinking in a DRb server running in the master because it must
> bind in some TCP port.
DRb can bind to UNIX domain sockets, too. But ask yourself:
Is your app really so broken that you always need DRb running on it?
You may also want to checkout Hijack, I haven't used/needed it myself,
but it could be interesting if an app is really in a lot of trouble:
http://www.rubyinside.com/hijack-get-a-live-irb-prompt-for-any-existing-ruby-process-2232.html
> The problem is that I should stop the DRb server before master reexec to avoid
> EADDINUSE. Well, I could code some rescue block...
rescuing EADDRINUSE in a loop is also how the "listen" directive
implements the :tries/:delay parameters.
> > $ rackup -h
> ...
> > -D, --daemonize run daemonized in the background
>
> Why "daemonize" is not present in unicornf configuration file?
Mainly it was cleaner and easier to test/implement it the way it is now.
It's also easier for somebody testing a setup to toggle it from the
command line.
I actually didn't want to support --daemonize at all since setsid(8),
is standard most GNU/Linux distros nowadays, but we support more than
just GNU/Linux.
> Really thanks a lot for all your help.
No problem.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-24 19:34 ` Eric Wong
@ 2009-12-25 22:09 ` Iñaki Baz Castillo
2009-12-26 1:30 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-25 22:09 UTC (permalink / raw)
To: mongrel-unicorn
El Jueves, 24 de Diciembre de 2009, Eric Wong escribió:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> > However I'm building my app now and when start it it usually has errors,
> > and that's good as it means I'm working :)
> > But sincerelly I would like to see the error just once rather than many
> > times due to workers respawn :)
> >
> > Also, is there any "interval" parameter so a worker is not re-spawned
> > until such timeout (after die).
>
> You can sleep in the before_fork hook.
You mean "before_exec", rigth? :)
Yes, it's the solution :)
> I actually didn't want to support --daemonize at all since setsid(8),
> is standard most GNU/Linux distros nowadays, but we support more than
> just GNU/Linux.
AFAIK setsid is not 100% valid to daemonize a process because:
- It doesn't handle the exit return code of the process execution (it just
returns 0 even if the process returns 1 at startup).
- It remains writting in the current stdout (the terminal in which you run
"setsid program_name".
Thanks a lot.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-23 7:26 ` Eric Wong
2009-12-23 10:22 ` Iñaki Baz Castillo
@ 2009-12-25 23:58 ` Iñaki Baz Castillo
1 sibling, 0 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-25 23:58 UTC (permalink / raw)
To: mongrel-unicorn
El Miércoles, 23 de Diciembre de 2009, Eric Wong escribió:
> > Usually if a worker (all the workers) fail to start at the moment of
> > running it, it obviously means that there is some error in the
> > application with prevents it to run. It could be great if Unicorn could
> > detect it.
>
> I'm generally hesitant to introduce code/features/bloat that aren't
> helpful to people who properly test configurations before deploying :)
>
> Adding this code doesn't solve problems. Either way the site can become
> inaccessible/unusable and problems are noticeable very quickly.
Ok, I should explain better why I need this feature:
I'm not coding a HTTP app on top of Unicorn to host in a server. Instead I'm
building a XCAP server (RFC 4825) using Rack and Unicorn. I'll also publish a
Ruby gem containing all the server (Unicorn + Rack application).
So, any user could install the gem, set the database(s), fill the
configuration file and star the server in background by running the provided
init script ("/etc/init.d/xcapserver start") which runs the server daemonized.
Imagine that the user wrote a typo, i.e:
after_fork do |server, worker|
# Note the space typo:
worker.u ser("www-data", "www-data") if Process.euid == 0
end
Then the init script would return 0 (success) but the server wouldn't work.
The user would check "ps aux | grep xcapserver" and would see the xcapserver
running. It would be complex to understand why the server it's failing.
So it would be great the following:
- The user start the init script (which calls "unicorn -D ...") with a
hypotethical new option ("--no-error-time 2").
- The command "unicorn -D" remains in foreground for 2 seconds before master
process going to background.
- Unicorn fails to create the workers at the moment (so within 2 seconds).
- Instead of re-spawning them, the command (still in foreground) exits with
return code 1 (error).
But there is other case which is much wrose IMHO. Imagine the user writes a
space typo:
stde rr_path "/var/log/xcapserver.err.log"
or imagine it uses a path that doesn't exist (/var/log/xcapserver/ doesn't
exist):
stderr_path "/var/log/xcapserver/err.log"
In both cases "unicorn -D" returns 0 but the server is not running (no unicorn
process running). So, why did it return 0? It's not an error when creating the
workers, but when running also the master process. IMHO in this case Unicorn
should return non zero (even if it called with "-D").
I'm playing right now with unicorn/launcher.rb (daemonize! method)but I get
nothing. Unfortunatelly I think that the usage of fork makes very difficult
the main command to behave as I desire (exit status).
If you could give me some tips I'd try to implement it.
Reall thanks a lot again and again :)
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-25 22:09 ` Iñaki Baz Castillo
@ 2009-12-26 1:30 ` Iñaki Baz Castillo
0 siblings, 0 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-26 1:30 UTC (permalink / raw)
To: mongrel-unicorn
El Viernes, 25 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> AFAIK setsid is not 100% valid to daemonize a process because:
>
> - It doesn't handle the exit return code of the process execution (it just
> returns 0 even if the process returns 1 at startup).
>
> - It remains writting in the current stdout (the terminal in which you run
> "setsid program_name".
Forget my last comment as the redirection must be done for the program to run.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
@ 2009-12-26 4:29 Iñaki Baz Castillo
2009-12-26 6:16 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-26 4:29 UTC (permalink / raw)
To: mongrel-unicorn
El Sábado, 26 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> I'm playing right now with unicorn/launcher.rb (daemonize! method)but I
> get nothing. Unfortunatelly I think that the usage of fork makes very
> difficult the main command to behave as I desire (exit status).
>
> If you could give me some tips I'd try to implement it.
I've implemented it! :)
If you are interested please review this file (I cannot attach it in the mail
as it's rejected by the maillist):
http://oversip.net/public/min_time_running.rb
It contains a modification for bin/unicorn and a rewritten
lib/unicorn/launcher.rb.
The code works for me for the two cases I explained in my previous mail.
Of course it's an optional feature (I've implemented it with "-M --min-
running-time SECONDS" in the options parser).
Regards.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-26 4:29 "unicorn -D" always returns 0 "success" (even when failed to load) Iñaki Baz Castillo
@ 2009-12-26 6:16 ` Eric Wong
2009-12-26 15:47 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-26 6:16 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Sábado, 26 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> > I'm playing right now with unicorn/launcher.rb (daemonize! method)but I
> > get nothing. Unfortunatelly I think that the usage of fork makes very
> > difficult the main command to behave as I desire (exit status).
> >
> > If you could give me some tips I'd try to implement it.
>
> I've implemented it! :)
>
> If you are interested please review this file (I cannot attach it in the mail
> as it's rejected by the maillist):
>
> http://oversip.net/public/min_time_running.rb
>
> It contains a modification for bin/unicorn and a rewritten
> lib/unicorn/launcher.rb.
Interesting, I could be tempted to accept this with a few changes...
Process.detach() spawns a background Thread. Having running threads
before forking won't fly with certain OS threading libraries (some
versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
What you're trying to accomplish does not require threads.
Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
of being non-portable for certain signals. Likewise with Process.kill
(0 being the only exception, of course).
I would trap() in the parent before any forking, in case the
sleep time is ever so low there's a race condition. Probably no
need to handle USR1, either, just let the parent die on its own,
or alternatively, have the parent sleep forever in case something
unforseen happens in the children...
> The code works for me for the two cases I explained in my previous mail.
> Of course it's an optional feature (I've implemented it with "-M --min-
> running-time SECONDS" in the options parser).
I believe the nginx grace period is hard-coded at 5 seconds, which should
be enough.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-26 6:16 ` Eric Wong
@ 2009-12-26 15:47 ` Iñaki Baz Castillo
2009-12-26 18:23 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-26 15:47 UTC (permalink / raw)
To: mongrel-unicorn
El Sábado, 26 de Diciembre de 2009, Eric Wong escribió:
> > http://oversip.net/public/min_time_running.rb
> >
> > It contains a modification for bin/unicorn and a rewritten
> > lib/unicorn/launcher.rb.
>
> Interesting, I could be tempted to accept this with a few changes...
>
> Process.detach() spawns a background Thread. Having running threads
> before forking won't fly with certain OS threading libraries (some
> versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
Yes, but there is a problem:
Imagine I don't use Process.detach and the master process ends when loads the
rack application (due to a typo or due to an error in unicorn conf file).
Then the master process would remain visible/alive but as zombie.
If so, when the controller process sends signal 0 to check the master process
it will always receive 'true' (a zombie process is a process which can receive
signals, it's does exist).
If I could check if the master process is zombie then I wouldn't need
Process.detach, but I don't know how to check if a process is zombie, no way
in:
http://ruby-doc.org/core/classes/Process.html
http://ruby-doc.org/core/classes/Process/Status.html
Any suggestion here?
> Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
> of being non-portable for certain signals. Likewise with Process.kill
> (0 being the only exception, of course).
Sure, I'll change it.
> I would trap() in the parent before any forking, in case the
> sleep time is ever so low there's a race condition.
Ok.
> Probably no need to handle USR1, either, just let the parent die on its own,
Note that the parent sleeps for 'min_running_time' + 1 (to avoid exiting
before receiving USR1 or USR2 from the controller process). But if the
controller process sends USR1 then the parent can already exit without waiting
one second more.
Well, this is not very interesting, but I think it's good in order to get a
faster exit code (it's not a good idea that a daemon takes long time to start
and return ans exit status).
> or alternatively, have the parent sleep forever in case something
> unforseen happens in the children...
But then the parent process doesn't end and continues in foreground. This is
not valid for running as daemon. The parent process must exit with the
appropriate status code after a few seconds to behave as any service running
in background.
> > The code works for me for the two cases I explained in my previous mail.
> > Of course it's an optional feature (I've implemented it with "-M --min-
> > running-time SECONDS" in the options parser).
>
> I believe the nginx grace period is hard-coded at 5 seconds, which should
> be enough.
Sorry, I don't know ngix. I need to use other reverse proxy (Pound) but let me
open a new thread about it :)
Anyhow I don't understand what you mean with "nginx grace period". How does
affect to Unicorn? Do you mean that, in case of implementing the above,
Unicorn wouldn't need a --min-running-time option? (I expect it would be
required since Unicorn is 100% independent of ngix)
Really thanks a lot. I'll do the changes you suggest and show you again (but
still don't know how to avoid Process.detach for now...).
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-26 15:47 ` Iñaki Baz Castillo
@ 2009-12-26 18:23 ` Iñaki Baz Castillo
2009-12-27 1:31 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-26 18:23 UTC (permalink / raw)
To: mongrel-unicorn
El Sábado, 26 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> El Sábado, 26 de Diciembre de 2009, Eric Wong escribió:
> > > http://oversip.net/public/min_time_running.rb
> > >
> > > It contains a modification for bin/unicorn and a rewritten
> > > lib/unicorn/launcher.rb.
> >
> > Interesting, I could be tempted to accept this with a few changes...
> >
> > Process.detach() spawns a background Thread. Having running threads
> > before forking won't fly with certain OS threading libraries (some
> > versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
>
> Yes, but there is a problem:
> Imagine I don't use Process.detach and the master process ends when loads
> the rack application (due to a typo or due to an error in unicorn conf
> file). Then the master process would remain visible/alive but as zombie.
> If so, when the controller process sends signal 0 to check the master
> process it will always receive 'true' (a zombie process is a process which
> can receive signals, it's does exist).
Hi, I've improved it a bit with your suggestions and also simplified it a lot.
Now there is no a "controller process". Instead, if "Unicorn.run" raises then
master process rescues the exception and sends USR1 to parent process so it
exists with 1.
So, Process.detach is not used anymore :)
Please check it from here:
http://oversip.net/public/unicorn_addons.rb
> > Use trap(Symbol), trap(Integer) is harder to read and has a likelyhood
> > of being non-portable for certain signals. Likewise with Process.kill
> > (0 being the only exception, of course).
Done in new code.
> > I would trap() in the parent before any forking, in case the
> > sleep time is ever so low there's a race condition.
Done in new code.
> > Probably no need to handle USR1, either, just let the parent die on its
> > own,
Now just one signal plays: USR1. It is sent by master process to parent
process if Unicorn.run raised an exception.
Regards.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-26 18:23 ` Iñaki Baz Castillo
@ 2009-12-27 1:31 ` Eric Wong
2009-12-27 3:06 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-27 1:31 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Sábado, 26 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> > El Sábado, 26 de Diciembre de 2009, Eric Wong escribió:
> > > Interesting, I could be tempted to accept this with a few changes...
> > >
> > > Process.detach() spawns a background Thread. Having running threads
> > > before forking won't fly with certain OS threading libraries (some
> > > versions of FreeBSD, I believe, check MRI Redmine for some open bugs).
> >
> > Yes, but there is a problem:
> > Imagine I don't use Process.detach and the master process ends when loads
> > the rack application (due to a typo or due to an error in unicorn conf
> > file). Then the master process would remain visible/alive but as zombie.
> > If so, when the controller process sends signal 0 to check the master
> > process it will always receive 'true' (a zombie process is a process which
> > can receive signals, it's does exist).
>
> Hi, I've improved it a bit with your suggestions and also simplified
> it a lot. Now there is no a "controller process". Instead, if
> "Unicorn.run" raises then master process rescues the exception and
> sends USR1 to parent process so it exists with 1.
Hi, I realized Unicorn.run inside the daemonize method doesn't work.
The daemonize method is also used by Rainbows! (and Zbatery).
However, I've reimplemented much of your idea here so it does not
involve sleeping at all, but rather shares a pipe with the grandparent
(started by the user) and Unicorn master process. The added benefit of
using a pipe is that there's no fuzzy sleeping involved at or grace
periods to worry about, too.
Also pushed out to the "ready_pipe" branch of
git://git.bogomips.org/unicorn.git
Let me know how it goes.
If everything looks good, I'll consider merging for 0.96.0.
>From 5eea32764571b721cd1a89cf9ebfa853c621ac9e Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Sat, 26 Dec 2009 17:04:57 -0800
Subject: [PATCH] exit with failure if master dies when daemonized
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This behavior change also means our grandparent (launched
from a controlling terminal or script) will wait until
the master process is ready before returning.
Thanks to Iñaki Baz Castillo for the initial implementations
and inspiration.
---
bin/unicorn | 2 +-
bin/unicorn_rails | 2 +-
lib/unicorn.rb | 9 +++++++--
lib/unicorn/launcher.rb | 37 +++++++++++++++++++++++++++++--------
test/exec/test_exec.rb | 2 +-
5 files changed, 39 insertions(+), 13 deletions(-)
diff --git a/bin/unicorn b/bin/unicorn
index 651c2ff..5af021d 100755
--- a/bin/unicorn
+++ b/bin/unicorn
@@ -161,5 +161,5 @@ if $DEBUG
})
end
-Unicorn::Launcher.daemonize! if daemonize
+Unicorn::Launcher.daemonize!(options) if daemonize
Unicorn.run(app, options)
diff --git a/bin/unicorn_rails b/bin/unicorn_rails
index 4a22a8c..b1458fc 100755
--- a/bin/unicorn_rails
+++ b/bin/unicorn_rails
@@ -202,6 +202,6 @@ end
if daemonize
options[:pid] = rails_pid
- Unicorn::Launcher.daemonize!
+ Unicorn::Launcher.daemonize!(options)
end
Unicorn.run(app, options)
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index 71d5994..ae05f03 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -25,7 +25,8 @@ module Unicorn
class << self
def run(app, options = {})
- HttpServer.new(app, options).start.join
+ ready_pipe = options.delete(:ready_pipe)
+ HttpServer.new(app, options).start.join(ready_pipe)
end
end
@@ -313,7 +314,7 @@ module Unicorn
# (or until a termination signal is sent). This handles signals
# one-at-a-time time and we'll happily drop signals in case somebody
# is signalling us too often.
- def join
+ def join(ready_pipe = nil)
# this pipe is used to wake us up from select(2) in #join when signals
# are trapped. See trap_deferred
init_self_pipe!
@@ -324,6 +325,10 @@ module Unicorn
trap(:CHLD) { |sig_nr| awaken_master }
proc_name 'master'
logger.info "master process ready" # test_exec.rb relies on this message
+ if ready_pipe
+ ready_pipe.syswrite($$.to_s)
+ ready_pipe.close
+ end
begin
loop do
reap_all_workers
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 1229b84..2d6ad97 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -19,21 +19,42 @@ class Unicorn::Launcher
# the directory it was started in when being re-executed
# to pickup code changes if the original deployment directory
# is a symlink or otherwise got replaced.
- def self.daemonize!
+ def self.daemonize!(options = {})
$stdin.reopen("/dev/null")
+ $stdin.sync = true # may not do anything...
# We only start a new process group if we're not being reexecuted
# and inheriting file descriptors from our parent
unless ENV['UNICORN_FD']
- exit if fork
- Process.setsid
- exit if fork
+ # grandparent - reads pipe, exits when master is ready
+ # \_ parent - exits immediately ASAP
+ # \_ unicorn master - writes to pipe when ready
- # $stderr/$stderr can/will be redirected separately in the Unicorn config
- Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
- Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+ rd, wr = IO.pipe
+ grandparent = $$
+ if fork
+ wr.close # grandparent does not write
+ else
+ rd.close # unicorn master does not read
+ Process.setsid
+ exit if fork # parent dies now
+ end
+
+ if grandparent == $$
+ # this will block until HttpServer#join runs (or it dies)
+ master_pid = rd.sysread(16).to_i
+ exit!(1) unless master_pid > 1
+ exit 0
+ else # unicorn master process
+ options[:ready_pipe] = wr
+ # $stderr/$stderr can/will be redirected separately in the
+ # Unicorn config
+ Unicorn::Configurator::DEFAULTS[:stderr_path] = "/dev/null"
+ Unicorn::Configurator::DEFAULTS[:stdout_path] = "/dev/null"
+
+ # returns so Unicorn.run can happen
+ end
end
- $stdin.sync = $stdout.sync = $stderr.sync = true
end
end
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index fc0719b..24ba856 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -805,7 +805,7 @@ EOF
exec($unicorn_bin, "-D", "-l#{@addr}:#{@port}", "-c#{ucfg.path}")
end
pid, status = Process.waitpid2(pid)
- assert status.success?, "original process exited successfully"
+ assert ! status.success?, "original process exited successfully"
sleep 1 # can't waitpid on a daemonized process :<
assert err.stat.size > 0
end
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-27 1:31 ` Eric Wong
@ 2009-12-27 3:06 ` Iñaki Baz Castillo
2009-12-27 3:07 ` Iñaki Baz Castillo
2009-12-28 3:29 ` Eric Wong
0 siblings, 2 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-27 3:06 UTC (permalink / raw)
To: mongrel-unicorn
El Domingo, 27 de Diciembre de 2009, Eric Wong escribió:
> > Hi, I've improved it a bit with your suggestions and also simplified
> > it a lot. Now there is no a "controller process". Instead, if
> > "Unicorn.run" raises then master process rescues the exception and
> > sends USR1 to parent process so it exists with 1.
>
> Hi, I realized Unicorn.run inside the daemonize method doesn't work.
Strange, it works for me...
> However, I've reimplemented much of your idea here so it does not
> involve sleeping at all, but rather shares a pipe with the grandparent
> (started by the user) and Unicorn master process. The added benefit of
> using a pipe is that there's no fuzzy sleeping involved at or grace
> periods to worry about, too.
>
> Also pushed out to the "ready_pipe" branch of
> git://git.bogomips.org/unicorn.git
>
> Let me know how it goes.
>
> If everything looks good, I'll consider merging for 0.96.0.
It's really awesome! I've tested it and it works great, and avoids the
sleeping stuff and a new commandlline option. Congratulations :)
However there is still a not solved case. Let me please explain it with two
examples:
1) In "after_fork" I set:
worker.user("non_existing_user","non_existing_group")
The master process would start properly and would notify "success" to
grandparent. (so the init script returns 0). But the fact is that all the
workers fail to start and are respawned again and again.
2) I use "Clogger" in my Rack config.ru. But I set a log file in a path that
doesn't exist. Clogger raises due to this (it doesn't try to create the full
path).
Again the master process has notified "success" to grandparent (exis status 0
then) but the workers are respawned again and again.
In both cases, if "preload_app" is false then doing a "ps" I see new master
processes being created by first master process (with new workers) and all of
them die while new ones born.
If "preload_app" is false then there are no borning master processes as just
the workers fail to start when loading the Rack app.
It would be great if the same mechanims (master notifies to grandparent) would
be implemented between first process loading the Rack app (a worker if
"preload_app") and master process, so if the first worker fails when loading
the Rack app then master process doesn't notify "success" to grandparent and
exists (instead of respawning again and again the workers).
If "preload_app" is true then I don't understand why your new code doesn't
work right now. I expect that master process notifies "success" to grandparent
before loading the Rack app, am I right? would make sense master process to
wait until Rack app is loaded and if it fails then notify "error" and exit
completely?
Kind regards and thanks a lot for your great work.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-27 3:06 ` Iñaki Baz Castillo
@ 2009-12-27 3:07 ` Iñaki Baz Castillo
2009-12-28 3:29 ` Eric Wong
1 sibling, 0 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-27 3:07 UTC (permalink / raw)
To: mongrel-unicorn
El Domingo, 27 de Diciembre de 2009, Iñaki Baz Castillo escribió:
> In both cases, if "preload_app" is false then doing a "ps" I see new
> master processes being created by first master process (with new workers)
> and all of them die while new ones born.
Here I meant:
if "preload_app" is true ...
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-27 3:06 ` Iñaki Baz Castillo
2009-12-27 3:07 ` Iñaki Baz Castillo
@ 2009-12-28 3:29 ` Eric Wong
2009-12-28 10:39 ` Iñaki Baz Castillo
2009-12-28 12:50 ` Iñaki Baz Castillo
1 sibling, 2 replies; 25+ messages in thread
From: Eric Wong @ 2009-12-28 3:29 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Domingo, 27 de Diciembre de 2009, Eric Wong escribió:
> > > Hi, I've improved it a bit with your suggestions and also simplified
> > > it a lot. Now there is no a "controller process". Instead, if
> > > "Unicorn.run" raises then master process rescues the exception and
> > > sends USR1 to parent process so it exists with 1.
> >
> > Hi, I realized Unicorn.run inside the daemonize method doesn't work.
>
> Strange, it works for me...
"Doesn't work" as in it's not friendly to servers based on Unicorn (like
Rainbows!).
> > However, I've reimplemented much of your idea here so it does not
> > involve sleeping at all, but rather shares a pipe with the grandparent
> > (started by the user) and Unicorn master process. The added benefit of
> > using a pipe is that there's no fuzzy sleeping involved at or grace
> > periods to worry about, too.
> >
> > Also pushed out to the "ready_pipe" branch of
> > git://git.bogomips.org/unicorn.git
> >
> > Let me know how it goes.
> >
> > If everything looks good, I'll consider merging for 0.96.0.
>
> It's really awesome! I've tested it and it works great, and avoids the
> sleeping stuff and a new commandlline option. Congratulations :)
The current version is actually slightly buggy in that it leaks
the pipe descriptor...
> However there is still a not solved case. Let me please explain it with two
> examples:
>
> 1) In "after_fork" I set:
> worker.user("non_existing_user","non_existing_group")
>
> The master process would start properly and would notify "success" to
> grandparent. (so the init script returns 0). But the fact is that all the
> workers fail to start and are respawned again and again.
For that particular case there'll be a Unicorn::Configurator#user
directive.
But really, there's absolutely no good reason to use user switching in a
backend application server like Unicorn.
I only added that feature to support derivative servers like Rainbows!,
and even then it's debatable since using things like iptables or load
balancers can be used to redirect port 80 to arbitrary ports anyways.
> 2) I use "Clogger" in my Rack config.ru. But I set a log file in a path that
> doesn't exist. Clogger raises due to this (it doesn't try to create the full
> path).
> Again the master process has notified "success" to grandparent (exis status 0
> then) but the workers are respawned again and again.
There's really an infinite number of ways to screw things up badly in
workers and cause them to flap. It's not possible to protect careless
users from all of them, so attempting to do so would be a waste of
effort.
Avoiding user-switching in an app server is a great first step, as it
eliminates an entire class of problems :)
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 3:29 ` Eric Wong
@ 2009-12-28 10:39 ` Iñaki Baz Castillo
2009-12-28 12:50 ` Iñaki Baz Castillo
1 sibling, 0 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-28 10:39 UTC (permalink / raw)
To: mongrel-unicorn
El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > The master process would start properly and would notify "success" to
> > grandparent. (so the init script returns 0). But the fact is that all the
> > workers fail to start and are respawned again and again.
>
> For that particular case there'll be a Unicorn::Configurator#user
> directive.
>
> But really, there's absolutely no good reason to use user switching in a
> backend application server like Unicorn.
>
> I only added that feature to support derivative servers like Rainbows!,
> and even then it's debatable since using things like iptables or load
> balancers can be used to redirect port 80 to arbitrary ports anyways.
Well, chaning the running user it's common in most of servers. I've already
found lots of cases of attacks to Apache servers running some "cool" PHP
application (so we get exploits in /tmp or/var/tmp as they are the only
writable paths for "www-data" user running apache).
However it's true that Unicorn approach (worker.user) is different as the
master process remains as root (but since the master process doesn't listen it
shouldn't matter).
So, do you mean that there will be a new configuration option called "user"
(and "group") so also themaster process would run as such user?
Thanks.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 3:29 ` Eric Wong
2009-12-28 10:39 ` Iñaki Baz Castillo
@ 2009-12-28 12:50 ` Iñaki Baz Castillo
2009-12-28 19:25 ` Eric Wong
1 sibling, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-28 12:50 UTC (permalink / raw)
To: mongrel-unicorn
El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > It's really awesome! I've tested it and it works great, and avoids the
> > sleeping stuff and a new commandlline option. Congratulations :)
>
> The current version is actually slightly buggy in that it leaks
> the pipe descriptor...
I've detected some other "issue":
Imagine port 80 is used by other application (as apache) and Unicorn is
configured to also bind in same port.
When running it in foreground all is great:
ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
ERROR -- : retrying in 3 seconds (1 tries left)
ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
ERROR -- : retrying in 3 seconds (0 tries left)
ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in
`initialize': Address already in use - bind(2) (Errno::EADDRINUSE)
But when running in background an ugly error is displayed:
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in `sysread':
end of file reached (EOFError)
from /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in
`daemonize!'
In both cases $? is 1 (error) but in the last case the error message is not
very useful.
I suspect what is happening: the master tries several times to bind and after
N retries it terminates (so it closes the pipe and grandparent gets EOFError).
As a suggestion, could the grandparent rescue such exception and display some
kind of error message?:
"The master couldn't be started. Inspect the log error or run in foreground"
Regards.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 12:50 ` Iñaki Baz Castillo
@ 2009-12-28 19:25 ` Eric Wong
2009-12-28 20:17 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-28 19:25 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > > It's really awesome! I've tested it and it works great, and avoids the
> > > sleeping stuff and a new commandlline option. Congratulations :)
> >
> > The current version is actually slightly buggy in that it leaks
> > the pipe descriptor...
>
> I've detected some other "issue":
>
> Imagine port 80 is used by other application (as apache) and Unicorn is
> configured to also bind in same port.
>
>
> When running it in foreground all is great:
>
> ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
> ERROR -- : retrying in 3 seconds (1 tries left)
> ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
> ERROR -- : retrying in 3 seconds (0 tries left)
> ERROR -- : adding listener failed addr=0.0.0.0:80 (in use)
> /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in
> `initialize': Address already in use - bind(2) (Errno::EADDRINUSE)
>
>
> But when running in background an ugly error is displayed:
>
> /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in `sysread':
> end of file reached (EOFError)
> from /usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/launcher.rb:45:in
> `daemonize!'
>
>
> In both cases $? is 1 (error) but in the last case the error message is not
> very useful.
>
> I suspect what is happening: the master tries several times to bind and after
> N retries it terminates (so it closes the pipe and grandparent gets EOFError).
>
> As a suggestion, could the grandparent rescue such exception and display some
> kind of error message?:
>
> "The master couldn't be started. Inspect the log error or run in foreground"
Thanks, pushed out to the ready_pipe branch of unicorn.git
>From 52eee4e424198a3c80793ee9c930fd3bb0285168 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Mon, 28 Dec 2009 11:16:00 -0800
Subject: [PATCH] launcher: descriptive error message on startup failure
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Rather than erroring out with a non-descript EOFError,
show a warning message telling users to check the logs
instead.
Reported-by: Iñaki Baz Castillo mid=200912281350.44760.ibc@aliax.net
---
lib/unicorn/launcher.rb | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/unicorn/launcher.rb b/lib/unicorn/launcher.rb
index 0ea836b..1871420 100644
--- a/lib/unicorn/launcher.rb
+++ b/lib/unicorn/launcher.rb
@@ -42,8 +42,11 @@ class Unicorn::Launcher
if grandparent == $$
# this will block until HttpServer#join runs (or it dies)
- master_pid = rd.readpartial(16).to_i
- exit!(1) unless master_pid > 1
+ master_pid = (rd.readpartial(16) rescue nil).to_i
+ unless master_pid > 1
+ warn "master failed to start, check stderr log for details"
+ exit!(1)
+ end
exit 0
else # unicorn master process
options[:ready_pipe] = wr
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 19:25 ` Eric Wong
@ 2009-12-28 20:17 ` Iñaki Baz Castillo
2009-12-28 20:32 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-28 20:17 UTC (permalink / raw)
To: mongrel-unicorn
El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > As a suggestion, could the grandparent rescue such exception and display
> > some kind of error message?:
> >
> > "The master couldn't be started. Inspect the log error or run in
> > foreground"
>
> Thanks, pushed out to the ready_pipe branch of unicorn.git
Hi, I attach a minor patch to make the error more verbose when Unicorn cannot
listen in a socket.
For example, as normal user (not root) I try to listen in a UNIX socket (ok)
and also on TCP port 84 (permission error). Actually I would see this error log:
I, [2009-12-28T21:05:31.503533 #26137] INFO -- : unlinking existing socket=/tmp/unicorn.sock
I, [2009-12-28T21:05:31.504019 #26137] INFO -- : listening on addr=/tmp/unicorn.sock fd=3
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in `initialize': Permission denied - bind(2) (Errno::EACCES)
By inspecting the error it seems that the problem is in the UNIX socket.
With my patch the log would be:
I, [2009-12-28T21:11:18.165918 #26589] INFO -- : unlinking existing socket=/tmp/openxdms.sock
I, [2009-12-28T21:11:18.166228 #26589] INFO -- : listening on addr=/tmp/openxdms.sock fd=3
F, [2009-12-28T21:11:18.166632 #26589] FATAL -- : error adding listener addr=0.0.0.0:84
/usr/local/lib/ruby1.9/site_ruby/1.9.1/unicorn/socket_helper.rb:110:in `initialize': Permission denied - bind(2) (Errno::EACCES)
I hope it's useful. Regards.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 20:17 ` Iñaki Baz Castillo
@ 2009-12-28 20:32 ` Eric Wong
2009-12-28 20:41 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-28 20:32 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > > As a suggestion, could the grandparent rescue such exception and display
> > > some kind of error message?:
> > >
> > > "The master couldn't be started. Inspect the log error or run in
> > > foreground"
> >
> > Thanks, pushed out to the ready_pipe branch of unicorn.git
>
> Hi, I attach a minor patch to make the error more verbose when Unicorn cannot
> listen in a socket.
Hi, it seems like you didn't attach the patch. Attaching patches is
strongly discouraged, though, inline patches are far easier to
review/edit/apply.
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 20:32 ` Eric Wong
@ 2009-12-28 20:41 ` Iñaki Baz Castillo
2009-12-29 0:17 ` Eric Wong
0 siblings, 1 reply; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-28 20:41 UTC (permalink / raw)
To: mongrel-unicorn
El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > > > As a suggestion, could the grandparent rescue such exception and
> > > > display some kind of error message?:
> > > >
> > > > "The master couldn't be started. Inspect the log error or run in
> > > > foreground"
> > >
> > > Thanks, pushed out to the ready_pipe branch of unicorn.git
> >
> > Hi, I attach a minor patch to make the error more verbose when Unicorn
> > cannot listen in a socket.
>
> Hi, it seems like you didn't attach the patch.
Ops, usually my mail client (Kmail) warns me when I write tha word "attach"
but attach nothing... not sure why it didn't it this time.
> Attaching patches is
> strongly discouraged, though, inline patches are far easier to
> review/edit/apply.
Ok, here it's (note that it's done over 'ready_pipe' branch):
--- unicorn.rb.orig 2009-12-28 21:02:47.000000000 +0100
+++ unicorn.rb 2009-12-28 21:11:12.000000000 +0100
@@ -307,6 +307,9 @@
"(#{tries < 0 ? 'infinite' : tries} tries left)"
sleep(delay)
retry
+ rescue => err
+ logger.fatal "error adding listener addr=#{address}"
+ raise err
end
end
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-28 20:41 ` Iñaki Baz Castillo
@ 2009-12-29 0:17 ` Eric Wong
2009-12-29 0:32 ` Iñaki Baz Castillo
0 siblings, 1 reply; 25+ messages in thread
From: Eric Wong @ 2009-12-29 0:17 UTC (permalink / raw)
To: unicorn list
Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > Iñaki Baz Castillo <ibc@aliax.net> wrote:
> > > El Lunes, 28 de Diciembre de 2009, Eric Wong escribió:
> > > > > As a suggestion, could the grandparent rescue such exception and
> > > > > display some kind of error message?:
> > > > >
> > > > > "The master couldn't be started. Inspect the log error or run in
> > > > > foreground"
> > > >
> > > > Thanks, pushed out to the ready_pipe branch of unicorn.git
> > >
> > > Hi, I attach a minor patch to make the error more verbose when Unicorn
> > > cannot listen in a socket.
> >
> > Hi, it seems like you didn't attach the patch.
>
> Ops, usually my mail client (Kmail) warns me when I write tha word "attach"
> but attach nothing... not sure why it didn't it this time.
>
> > Attaching patches is
> > strongly discouraged, though, inline patches are far easier to
> > review/edit/apply.
>
> Ok, here it's (note that it's done over 'ready_pipe' branch):
Thanks, applied with your name and pushed out to both ready_pipe and
master in unicorn.git
> --- unicorn.rb.orig 2009-12-28 21:02:47.000000000 +0100
> +++ unicorn.rb 2009-12-28 21:11:12.000000000 +0100
It should be easier to make a commit using git and then just
"git format-patch -M" to output it, proposed commit message and all.
In the HACKING doc, I've been recommending people follow the
SubmittingPatches document distributed with git when submitting
patches to Unicorn:
http://git.bogomips.org/cgit/mirrors/git.git/tree/Documentation/SubmittingPatches
--
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: "unicorn -D" always returns 0 "success" (even when failed to load)
2009-12-29 0:17 ` Eric Wong
@ 2009-12-29 0:32 ` Iñaki Baz Castillo
0 siblings, 0 replies; 25+ messages in thread
From: Iñaki Baz Castillo @ 2009-12-29 0:32 UTC (permalink / raw)
To: mongrel-unicorn
El Martes, 29 de Diciembre de 2009, Eric Wong escribió:
> Thanks, applied with your name and pushed out to both ready_pipe and
> master in unicorn.git
Thanks a lot :)
> > --- unicorn.rb.orig 2009-12-28 21:02:47.000000000 +0100
> > +++ unicorn.rb 2009-12-28 21:11:12.000000000 +0100
>
> It should be easier to make a commit using git and then just
> "git format-patch -M" to output it, proposed commit message and all.
>
> In the HACKING doc, I've been recommending people follow the
> SubmittingPatches document distributed with git when submitting
> patches to Unicorn:
>
> http://git.bogomips.org/cgit/mirrors/git.git/tree/Documentation/Submitting
> Patches
Ok, I'll read it.
Thanks againg.
--
Iñaki Baz Castillo <ibc@aliax.net>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2009-12-29 0:33 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-12-26 4:29 "unicorn -D" always returns 0 "success" (even when failed to load) Iñaki Baz Castillo
2009-12-26 6:16 ` Eric Wong
2009-12-26 15:47 ` Iñaki Baz Castillo
2009-12-26 18:23 ` Iñaki Baz Castillo
2009-12-27 1:31 ` Eric Wong
2009-12-27 3:06 ` Iñaki Baz Castillo
2009-12-27 3:07 ` Iñaki Baz Castillo
2009-12-28 3:29 ` Eric Wong
2009-12-28 10:39 ` Iñaki Baz Castillo
2009-12-28 12:50 ` Iñaki Baz Castillo
2009-12-28 19:25 ` Eric Wong
2009-12-28 20:17 ` Iñaki Baz Castillo
2009-12-28 20:32 ` Eric Wong
2009-12-28 20:41 ` Iñaki Baz Castillo
2009-12-29 0:17 ` Eric Wong
2009-12-29 0:32 ` Iñaki Baz Castillo
-- strict thread matches above, loose matches on Subject: below --
2009-12-23 2:20 Iñaki Baz Castillo
2009-12-23 7:26 ` Eric Wong
2009-12-23 10:22 ` Iñaki Baz Castillo
2009-12-23 20:35 ` Eric Wong
2009-12-24 9:30 ` Iñaki Baz Castillo
2009-12-24 19:34 ` Eric Wong
2009-12-25 22:09 ` Iñaki Baz Castillo
2009-12-26 1:30 ` Iñaki Baz Castillo
2009-12-25 23:58 ` Iñaki Baz Castillo
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).