Re: Issues with PID file renaming

[Eric Wong <normalperson@yhbt.net>]

Jimmy Soho <jimmy.soho@gmail.com> wrote:
> Hi Eric,
> 
> Since we upgraded from 4.6.2 to 4.7.0 we're regularly having issues
> where one or more unicorn master processes would not upgrade
> correctly, resulting in an (old) master process.
> 
> What I notice is the following: when upgrading with 4.6.2 the file
> unicorn.pid is copied to unicorn.pid.oldbin and the unicorn.pid file
> is updated (the (new) master process). After a while the worker pid
> files are updated and the unicorn.pid.oldbin file disappears, and all
> is fine.

Ugh.  This is what I feared...  Slow startup time of most Ruby web apps
doesn't help.

> When upgrading with 4.7.0 though the file unicorn.pid.oldbin is
> created, but there is no unicorn.pid file. After a while (when the new
> master process has finished initialising, and is ready to fork the
> workers) the worker pid files are updated and a unicorn.pid file
> appears, and then the unicorn.pid.oldbin file disappears.
> 
> So there is a relatively long period where there is no unicorn.pid file.
> 
> I think the problem for us is caused by monit, our process monitor,
> which monitors the unicorn.pid file:
> 
> check process unicorn with pidfile
> /srv/app.itrp-staging.com/shared/pids/unicorn.pid
>   start program = "/etc/init.d/unicorn start"
>   stop program = "/etc/init.d/unicorn stop"
>   ...

Is there an alternate way to monitor unicorn with monit?
But I'd like to keep your case working if possible.

> Because the unicorn.pid file is absent for a relatively long period
> monit will try to start unicorn, while an upgrade is in progress.
> (which might be another issue: the start action should recognise a
> start or upgrade process is already underway; sadly this check too
> relies on the existence of the unicorn.pid file)
> 
> I think the way 4.6.2 worked is better. There should be a pid file for
> the new master process the moment it's created.

> What do you think?

How about having the old process create a hard link to .oldbin,
and having the new one override the pid if Process.ppid == pid file?
The check is still racy, but that's what pid files are :<
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying