PID files not being removed on shutdown

PID files not being removed on shutdown

[Jon Wood]

Hello,

I'm quite consistently seeing an issue where Unicorn leaves its PID
files in place after a clean shut down, which is causing some issues
when deploying releases that update Unicorn config.

We're doing hot restarts with USR2+QUIT, and using Upstart for process
supervision based on the presence of a PID file (this admittedly might
be a mistake, but I've not seen any better solutions).

This works fine in the case of a hot restart since Unicorn correctly
updates the PID file, however when we change the configuration for
Unicorn shutting down the master process doesn't remove the PID file,
so the supervising Upstart script doesn't detect that the master has
been shut down and bring up a new one.

I've created a gist[1] of our Upstart and Unicorn configuration files.
As I say, we might just be doing something stupid here, but Google
doesn't seem to have shed any light on the issue.

Thanks for any help,
Jon

[1] https://gist.github.com/jellybob/3789c3f3a00989b63d74
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: PID files not being removed on shutdown

[Eric Wong]

Jon Wood <jon@hubbub.co.uk> wrote:
> Hello,
> 
> I'm quite consistently seeing an issue where Unicorn leaves its PID
> files in place after a clean shut down, which is causing some issues
> when deploying releases that update Unicorn config.
> 
> We're doing hot restarts with USR2+QUIT, and using Upstart for process
> supervision based on the presence of a PID file (this admittedly might
> be a mistake, but I've not seen any better solutions).

It's probably better to test the listening port unicorn uses
(perhaps setup a generic health check endpoint) via socat/curl
to detect if unicorn is down.

> This works fine in the case of a hot restart since Unicorn correctly
> updates the PID file, however when we change the configuration for
> Unicorn shutting down the master process doesn't remove the PID file,
> so the supervising Upstart script doesn't detect that the master has
> been shut down and bring up a new one.
> 
> I've created a gist[1] of our Upstart and Unicorn configuration files.
> As I say, we might just be doing something stupid here, but Google
> doesn't seem to have shed any light on the issue.

> [1] https://gist.github.com/jellybob/3789c3f3a00989b63d74

Do you have anything logging to stderr_path?  That might give you a
better idea of what's wrong.  Unfortunately, PID files are always bound
to be racy so perhaps supervisor scripts should test and endpoint of an
app using socat or curl.

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying