From: Graham Bleach <graham@darkskills.org.uk>
To: mongrel-unicorn@rubyforge.org
Subject: select(): Interrupted system call from curb when stopping unicorn
Date: Wed, 7 Nov 2012 16:42:37 +0000 [thread overview]
Message-ID: <CAJ4ZL52kKChaBcWFNnAbfiQ9BtHQ-0nZZ4jARJma2mcvcfv_iQ@mail.gmail.com> (raw)
Hi,
We've just migrated one of our rails applications from nginx/passenger
running on REE 1.8.7 Ubuntu 8.04 to unicorn running on MRI 1.9.3 on
Ubuntu 10.04. The app makes a number of calls to internal services
using curb.
Our deployment script stops unicorn by sending SIGQUIT to the unicorn
master, sleeps for a few seconds to ensure that HAProxy has taken the
node out of service and then starts unicorn again.
However, since the migration, we've started seeing a few errors on
each deploy, when the workers receive the QUIT from the master:
ERROR ( XXX) {"exception":{"class":"RuntimeError","message":"select():
Interrupted system
call","backtrace":["/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform'","/path/bundle/ruby/1.9.1/gems/curb-0.7.18/lib/curl/easy.rb:39:in
`perform'","/path/bundle/ruby/1.9.1/gems/songkick-transport-0.1.4/lib/songkick/transport/curb.rb:50:in
...
If we're reading this correctly, curb is inside a select() call, gets
interrupted by the QUIT signal and doesn't retry the select() and the
process terminates, causing an error for the unfortunate end user.
Our options for stopping our users seeing error pages seem to be:
1) Patch curb to retry the select() if errno = EINTR as per
https://github.com/taf2/curb/issues/117
We're not confident enough in our ability to do this properly, but
it's probably the most correct way to solve this.
2) Do something to make sure that requests have all completed before
sending the QUIT.
For example, we could change our deployment to make the health check
url that HAProxy polls start returning a non-200 return code, wait
until all connections have completed and then send the QUIT.
The main issue with this is that short of polling the HAProxy status
page, we're not entirely sure how to verify that all connections have
completed.
Does anyone have any suggested solutions or comments?
Regards,
Graham
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
next reply other threads:[~2012-11-07 16:42 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-07 16:42 Graham Bleach [this message]
2012-11-07 21:11 ` select(): Interrupted system call from curb when stopping unicorn Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJ4ZL52kKChaBcWFNnAbfiQ9BtHQ-0nZZ4jARJma2mcvcfv_iQ@mail.gmail.com \
--to=graham@darkskills.org.uk \
--cc=mongrel-unicorn@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).