unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <normalperson@yhbt.net>
To: unicorn list <mongrel-unicorn@rubyforge.org>
Subject: Re: Problem with binding UNIX listeners before checking PID
Date: Mon, 4 Oct 2010 04:17:13 +0000	[thread overview]
Message-ID: <20101004041713.GA28709@dcvr.yhbt.net> (raw)
In-Reply-To: <8D95A44B-A098-43BE-B532-7D74BD957F31@darkridge.com>

Jordan Ritter <jpr5@darkridge.com> wrote:
> Howdy.
> I have lately been frustrated by the following use case:
> 	1. Run nginx/unicorn in production, listening on a UNIX socket
> 	with a defined pid file.  Things run good.
> 	2. Someone pushes code, unicorn restarts just fine, workers are
> 	all up and running.
> 	3. But someone is suspicious, or maybe they forget which
> 	box they're logged into, so they invoke unicorn manually.
> 	Same directory, same settings.
> 	4. It looks like the pid file check kicked in, because unicorn
> 	refuses to boot - hey, it's already running, bugger off.  great.
> 	5. BUT, this happened *after* the listener processing: the
> 	manually-invoked unicorn unlinks the real unicorn master's unix
> 	listener, so it's left dead in the water and everybody loses.
> unicorn master doesn't know its listener is actually gone (but lsof shows
> open unix socket fd, netstat shows unix socket still present, so cursory
> investigation is misleading), but nginx keeps spewing ECONNREFUSEDs
> because the unix socket it's hitting belongs to that accidental unicorn
> instance that already decided not to stick around.
> I think this is effectively about a behavioral difference in
> Unicorn::SocketHelper#bind_listen around the handling of UNIX vs. TCP
> sockets (this doesn't happen with TCP sockets because there's no
> unlink/disconnect step), and the fact that HttpServer#start evaluates
> the listener config before the PID path/config.
> Now I see comments in and around HttpServer#initialize talking about races
> wrt binding to the listener and whatnot, and being newish to the codebase
> I admit I haven't yet fully absorbed all the considerations at play.
> But I think it's fair to say that killing the listener(s) (in the UNIX
> socket case) before discovering you shouldn't have run in the first place
> (from the PID file) qualifies as buggy/bad/broken behavior.

Hi Jordan,

Thanks for the detailed bug report.  I knew from experience with other
daemons that lingering UNIX sockets caused troubles for some users, but
I failed to take into account the case where a user mistakenly starting
the process twice.

Yes, getting pid file writing/ordering "right"[1] is very tricky.

> I might suggest simply swapping their processing order in #start, but
> given the complexity of in-place restarts and other race considerations,
> I have doubts solving this would be that easy.

That wouldn't work if pid files weren't in use at all.

> Any thoughts/ideas?

A simpler check would be to use connect(2) (but not make any HTTP request)
to see if the socket is alive.  Patch coming.

[1] - I don't believe there actually is a way to always be right,
      just less bad/broken than the alternatives.
Eric Wong
Unicorn mailing list - mongrel-unicorn@rubyforge.org
Do not quote signatures (like this one) or top post when replying

  reply	other threads:[~2010-10-04  4:26 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-10-02 16:38 Problem with binding UNIX listeners before checking PID Jordan Ritter
2010-10-04  4:17 ` Eric Wong [this message]
2010-10-04  4:22   ` [PATCH] avoid unlinking actively listening sockets Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20101004041713.GA28709@dcvr.yhbt.net \
    --to=normalperson@yhbt.net \
    --cc=mongrel-unicorn@rubyforge.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).